ORBIT 


VOLUME  24,  NUMBER  2, 1993 


A co-publication  of  the  Ontario 
Institute  for  Studies  in  Education 
and  Nelson  Canada 


Standards, 

Benchmarks 

and  Ontario 

Cnrrtcmum 


^SSAccormtability 


SPECIAL  ISSUE 


VOLUME  24,  NUMBER  2 1993 


©KBIT 


A co-publication  of  the  Ontario  Institute 
for  Studies  in  Education  and  Nelson  Canada 

EDITORIAL  OFFICES: 

ORBIT 

252  Bloor  Street  West 
Toronto,  Ontario,  M5S  1 V6 

EDITOR:  Heather  Berkeley 
assistant  EDITOR:  Donna  Hutchins 
consultants:  Ardra  Cole,  Jane  Hill,  David  E.  Hunt, 
Peter  McLaren,  Ruth  Scott,  and  Marielle  Simon 

GRAPHIC  DESIGN:  Helmut  W.  Weyerstrahs 
TYPESETTING:  Amheim  Typesetting  & Design 

Printed  and  bound  in  Canada 
by  Webcom 
on  recycled  paper 

SUBSCRIPTION  RATES: 

$32.10  per  subscription  year  (4  issues).  Incl.  7%  GST 
To  order:  Mail  cheque  or  money  order  to 
ORBIT 

P.O.  Box  10,  Station  F 
Toronto,  Ontario,  M4Y  2L4 

Phone:  (416)  696-2295 
Fax:  (416)  696-8409 

Second  Class  Mail  Registration  Number  4511 
ISSN  0030-4433  02 


NOTES  FROM  THE  DIRECTOR 


This  issue  of  Orbit,  guest-edited  by 
Professor  Howard  Russell  of  our 
Curriculum  Department,  deals  with  a 
topic  of  great  importance  to  students, 
parents,  and  educators  — accountability 
and  standards  and  measuring  student 
academic  achievement.  Its  contributors 
include  OISE  faculty  and  staff,  Ministry  of 
Education  and  Training  officials,  school 
board  administrators,  teachers,  and  a 
representative  of  the  business 
community. 

There  is  a thorough  discussion  of  one 
aspect  of  the  current  debate  on 
assessment  — namely,  the  development 
of  concrete  performance  standards 
against  which  an  individual  student’s 
efforts  can  be  compared,  an  approach 
which  diverges  from  more  traditional 
standardized  testing.  This  approach  has 
been  developed  over  the  past  five  to  ten 
years  by  the  Ministry  of  Education  and 
T raining  and  by  various  school  boards 
across  the  province.  Their  work  is  well 
represented  in  the  issue. 

The  issue  marks  the  continuation  of 
our  co-publication  agreement  with 
Nelson  Canada. 

Arthur  Kruger 
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Standards , Benchmarks, 
and  Ontario  Curriculum 


Introduction 


Benchmarks  and  standards  — which  have 
arrived  in  Ontario  schools  during  the  past 
decade  — constitute  a response  to  the 
continuing  pressure  for  public  account- 
ability in  education  and  are  slowly  gain- 
ing acceptance  among  teacher  groups, 
trustees,  parents,  business  officials,  and 
governments.  Although  there  are  many 
definitions  for  benchmarks  and  for  stan- 
dards, they  all  fit  in  the  general  category 
of  educational  indicators  of  student 
achievement. 

In  its  use  in  education,  a benchmark  is 
a standard  of  performance  against  which 
students’  progress  and  achievement  can 
be  compared.  To  use  more  commonplace 
terminology,  the  Ontario  Ministry  of  Edu- 
cation and  Training  has  adopted  the  term 
“standards”  to  replace  “benchmarks,”  and 
to  indicate  continuity  and  avoid  further 
confusion,  now  uses  the  term  “standards 
(benchmarks).”  In  this  issue  of  Orbit,  we 
use  the  terms  standards  and  benchmarks 
interchangeably  except  where  boards  of 
education  specifically  use  the  term 
“benchmark”  only. 

Benchmarks  and  standards  focus  on 
curriculum  substance  in  an  open  manner 
that  permits,  and  perhaps  encourages, 
public  debate  about  what  students  are 
taught  and  what  they  learn  as  well  as 
what  general  directions  education  should 
take.  The  more  traditional  indicators  have 
focussed  on  marks  and  general  scores 
which  cannot  be  related  to  changes  need- 
ed in  curricula  or  programs. 

This  issue  of  Orbit  begins  with  a sec- 
tion on  the  Ontario  Ministry  of  Educa- 
tion and  Training’s  initiatives.  The  lead 
article  by  Gail  Rappolt  focusses  attention 
on  the  Ontario  standards.  She  provides 
the  Ministry’s  background  discussion, 
principles,  and  process  in  the  develop- 
ment of  provincial  standards.  The  princi- 
ples upon  which  they  were  developed 
constitute  an  especially  important  start- 
ing point.  The  articles  by  Neil  Graham 
and  Gerry  Hurtubisc  tell  us  what  stu- 
dents arc  to  learn  and  teachers  arc  to 
teach  in  language  and  mathematics, 
respectively.  These  clear  accounts  of 


Howard  Russell 

Professor  of  Curriculum,  OISE 

outcomes  will  be  used  also  in  debates 
about  what  should  be  taught  in  our 
schools.  The  article  by  Marielle  Simon 
outlines  the  Ministry’s  benchmarks  ini- 
tiative in  French  and  describes  some 
local  initiatives  in  French-language 
school  boards. 

The  second  section  indicates  some  of 
the  perceptions  and  expectations  of  the 
public  and  of  the  business  community. 
Doug  Hart  presents  public  opinion  poll 
information  which  provides  readers  with 
the  best  available  evidence  on  what  the 
public  wants.  The  public  wants  to  know 
what  their  own  particular  children  have 
learned  and  they  want  the  teachers  of 
those  individual  children  to  have  a role  in 
determining  what  is  taught  and  how  stu- 
dents are  evaluated.  They  also  want  to 
know  what  impact  our  provincial  pro- 
grams have  on  student  learning  and  they 
want  to  be  able  to  make  comparisons  over 
time  and  across  provincial  and  national 
boundaries.  Our  authors  in  this  issue  of 
Orbit  have  addressed  these  questions. 

The  voice  of  the  business  community 
is  presented  through  an  interview  with 
Jim  Poole,  who  represents  the  Ontario 
Chamber  of  Commerce  on  various  com- 
mittees dealing  with  education.  His 
insights  into  the  issue  of  public  account- 
ability in  education  are  based  on  careful 
analysis  of  a vast  array  of  up-to-date 
information,  and  as  a consequence  they 
may  be  both  revealing  and  generally 
acceptable  to  educators. 

The  third  section  of  this  issue  deals 
with  the  Toronto  Board  of  Education’s 
benchmarks  Program.  It  begins  with  an 
interview  with  the  Director  of  Education, 
Joan  Green,  and  the  Superintendent  of 
Curriculum,  Marilyn  Sullivan.  We  have 
given  special  attention  to  the  Toronto  pro- 
gram because  it  has  a long  history  behind 
it  and  it  has  given  other  school  systems, 


both  here  and  abroad,  an  opportunity  to 
see  how  a diverse  curriculum  in  a cos- 
mopolitan city  can  be  monitored  effec- 
tively. Ms.  Green’s  comments  on  the 
acceptance  of  common  benchmarks 
across  racial  and  ethnic  lines  are  both 
helpful  and  insightful.  The  Donnelly  and 
Larter  article  and  the  Spielman  article 
spell  out  some  of  the  details  that  show  the 
nature  of  Toronto  benchmarks. 

The  fourth  section  explores  some  other 
initiatives  at  the  school  and  board  levels. 
The  first  article  is  based  on  a paper  Chris- 
tine Suurtamm  prepared  for  the  1992 
Annual  Meeting  of  the  Ontario  Educa- 
tional Research  Council.  It  covers  five 
boards  that  have  established  benchmark 
programs  and  it  outlines  some  of  their 
essential  features.  Betty  Lou  Borho  brings 
some  clarity  and  realism  to  what  may  be 
labelled  “real  mathematics”  in  the  prima- 
ry grades.  She  labels  her  benchmarks  the 
“s”  list.  The  “s”  signifies  “school”  and  it 
helps  to  distinguish  among  objectives  on 
the  basis  of  where  the  learning  takes 
place.  Stuart  Telfer  shows  how  bench- 
mark information  is  used  at  the  school 
board  level.  His  benchmarks  are  in  fact 
test  items  in  the  usual  sense,  and  he  pre- 
sents some  novel  ideas  for  connecting 
curriculum  change  to  benchmarks. 

The  issue  concludes  with  a discussion 
on  standards,  benchmarks,  and  curricu- 
lum with  Bruce  Archer,  Burle  Summers, 
Ken  Travers,  and  myself.  The  view  of 
benchmarks  and  standards  from  Wash- 
ington and  the  National  Science  Founda- 
tion, as  expressed  by  Ken  Travers,  indi- 
cates the  basic  similarity  between  the 
accountability  issues  we  face  in  Ontario 
and  those  faced  in  other  jurisdictions. 

In  a sense,  we  have  defined  bench- 
marks and  standards  by  giving  examples. 
The  rest  of  die  story  will  be  the  array  of 
uses  and  reporting  processes  dial  emerge 
over  time.  That  part  of  the  story  has  yet 
to  come.  We  arc  very  much  interested  in 
our  readers’  comments  on  the  substance 
of  this  issue  and  we  encourage  you  to 
write  to  us  cam  of  the  Editor  of  Orbit  at 
OISE. 
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Gail  Rappolt 

Field.  Co-ordinator 
Provincial  Benchmarks  Project, 

Ontario  Ministry  of  Education 
and  Training 

It  has  been  stated  that  the  1990s  will  be 
the  decade  of  accountability.  Nowhere 
will  that  be  more  apparent  than  in  educa- 
tion. A recent  article  in  The  Economist 
identified  the  quality  of  a county’s  edu- 
cation system  as  the  way  countries  will 
be  measured  in  the  90s.  It  went  on  to 
compare  student  achievement  in  the  90s 
to  GNP  in  the  70s  or  80s  as  a measure  of 
a country’s  international  prominence. 
The  combination  of  two  phenomena  — 
the  world-wide  focus  on  quality,  results, 
and  accountability  and  the  emphasis  on 
educating  all  of  society  to  a much  higher 
level  and  for  a quite  different  world  — 
has  put  tremendous  pressure  on  educa- 
tion systems,  schools,  and  educators. 

The  response  at  all  levels  has  been  to 
turn  much  more  attention  to  the  actual 
outcomes  we  want,  to  the  standards  we 
expect  and  the  student  results  we  are  get- 
ting, and  to  how  to  measure  and  report 
those  results.  Many  school  boards  in 
Ontario  began  work  on  such  projects  in 
the  late  1980s  or  early  1990s,  the  most 
widely  known  of  these  being  the  Toronto 
Benchmarks  Project.  In  the  fall  of  1991, 
the  Ministry  of  Education  began  a stan- 
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Developing  Provincial  Standards 


dards  project  as  well,  at  that  time  known 
as  the  Provincial  Benchmarks  Project. 
The  intention  was  to  draw  on  the  work 
already  done  in  England,  Scotland,  some 
provinces  in  Canada,  and  some  Ontario 
school  boards  to  develop  a comprehen- 
sive set  of  publicly  agreed  upon  stan- 
dards that  would  be  part  of  an  overall 
assessment  or  evaluation  framework. 

The  first  task  was  to  determine  the 
principles  or  beliefs  on  which  an  assess- 
ment system  should  be  based.  Over  time 
the  following  set  of  principles  emerged: 

Standards  referenced 

The  definition  of  the  standard  should 
come  before  the  assessment.  In  other 
words,  we  are  going  to  agree  on  a pre- 
determined standard  and  then  test  against 
it.  We  are  not  merely  going  to  test  and 
rank  results  against  each  other. 

Improvement  focussed 

The  primary  purpose  of  assessment  should 
be  to  improve  results  — for  the  student, 
the  school,  and  the  jurisdiction.  Although 
that  sounds  obvious,  many  of  the  large 
assessments  currently  conducted  do  not 
provide  either  the  type  or  the  quality  of 
information  that  leads  to  improvement; 
they  merely  provide  scores  to  rank  per- 
formers against  each  other  or  against  norm 
or  average. 

Fair  to  all 

Assessment  systems  must  be  sensitive  to 
class,  race,  culture,  and  gender  differences 


and  to  disabilities;  they  must  not  penalize 
any  group  in  subtle  or  covert  ways. 

Valid 

Assessment  systems  must  assess  the  out- 
comes that  are  worth  assessing  and  use 
appropriate  tools  to  do  so.  Implicit  in  this 
statement  is  the  recognition  that  multiple- 
choice  tests,  indeed  paper  and  pencil  tests 
alone,  cannot  assess  all  the  outcomes  of 
significance.  Embedded  in  the  principle  of 
validity  is  reliability:  to  be  reliable,  the 
assessments  need  to  yield  consistent 
results. 

Relevant 

Assessment  systems  must  be  used  and 
reported  in  the  context  of  other  relevant 
information.  For  example,  information  on 
student  results  taken  out  of  context  says 
little  about  the  quality  of  instruction  or 
value  for  money. 

Collaborative 

Parents  and  the  community  should  be 
involved  in  setting  the  standards.  Teachers 
should  be  involved  in  designing  the 
assessment  systems. 

Understandable 

Procedures  and  results  should  be  under- 
standable to  teachers  and  parents  as  well 
as  to  researchers  and  administrators. 
Results  should  be  public  and  reported 
against  pre-determined  standards.  Report- 
ing results  against  other  schools  or  juris- 
dictions leads  to  what  is  called  “high- 
stakes  testing”  and  undermines  several  of 
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the  other  principles  (standards  referenced, 
improvement  focussed,  valid,  reliable). 

Refined  regularly 

The  pace  of  change  in  all  aspects  of 
today’s  world  has  altered  traditional  views 
about  using  the  same  test  over  decades  for 
purposes  of  reliability.  Assessment  sys- 
tems and  standards  themselves  should  be 
subject  to  continuous  review  and  improve- 
ment. 

The  second  task  of  the  provincial  pro- 
ject was  to  develop  a framework  in 
which  standards  and  assessment  would 
fit  (see  Figure  1).  It  was  clear  that  the 
framework  had  to  reflect  the  results  or 
outcomes  focus  of  the  90s  and  to  differ- 
entiate between  outcomes  and  standards. 
There  were  several  major  shifts  in  think- 
ing that  had  to  occur  for  both  educators 
and  the  public. 

The  first  was  the  shift  from  what  we 
called  an  “intentions-based”  curriculum 
(in  Ontario  represented  by  the  Formative 
Years  and  OS:IS),  where  the  curriculum 
is  stated  in  terms  of  objectives  and  the 
teacher  is  the  person  responsible  for  the 
action.  Most  objectives  at  the  Ministry 
level  have  been  written  in  this  way  — 
for  example,  “students  will  have  oppor- 
tunities to  ...”  or  “students  will  experi- 
ence ...”  Here  the  onus  is  on  the  teacher 
to  act,  perform,  and  deliver  the  result.  In 
a shift  to  an  “outcomes-based”  system, 
the  student  is  now  accountable  for  the 
outcome  — for  example,  “students  will 
demonstrate  awareness  of  the  four  cue- 
ing systems  when  reading.”  Implicit  in 
this  student  performance  or  achievement 
is  the  teacher’s  role  in  ensuring  that  these 
skills  are  taught  and  learned;  however, 
the  student  must  take  the  actual  action 
and  demonstrate  his  or  her  achievement. 

The  second  shift  was  to  separate  the 
general  result  that  is  desired  (the  out- 
come) from  the  level  of  performance  that 
is  expected  (the  standard).  Whether  or 
not  there  are  two  separate  concepts  here 
is  still  hotly  debated  in  some  circles, 
though  researchers  like  Grant  Wiggins 
make  a very  clear  differentiation  (sec  his 
1991  article  “Standards,  Not  Standard- 
ization”). It  was  the  contention  of  the 
Provincial  Benchmarks  Project  that  there 
needed  to  be  two  quite  separate  concepts 
articulated.  We  further  suggested  that  lor 
now,  as  we  make  the  transition  from  an 
intentions-  and  subject-based  curriculum 
to  an  outcomes-based  and  cross-disci- 
plinary curriculum,  we  need  to  keep  sub- 


ject-based standards  as  a support  for  both 
teachers  and  parents. 

The  third  shift  was  to  demonstrate  that 
we  could  use  the  same  standards  of  per- 
formance for  measuring  student  achieve- 
ment in  the  classroom  as  we  would  use  to 
monitor  programs  in  schools,  school 
boards,  or  the  province.  Previous  assess- 
ment frameworks  had  kept  these  two 
types  of  assessment  separate  partly  to 
avoid  high-stakes  testing  and  partly 
because  the  large-scale  achievement  tests 
had  not  tended  to  measure  complex  con- 
cepts, higher-level  thinking  skills,  or  atti- 
tudes. What  made  this  idea  so  difficult  to 

Figure  1 Accountability  Flowchart 


accept  was  that  people  did  not  differenti- 
ate between  standards  and  standardized 
(tests).  In  addition,  we  had  to  demon- 
strate that  these  “standards”  were  accept- 
able against  international  standards. 
Responding  to  a demand  for  international 
comparability  while  attempting  to 
measure  the  complex  skills  and  perfor- 
mance that  are  demanded  in  the  Common 
Curriculum  will  continue  to  be  a very 
challenging  task.  In  their  recently  re- 
leased book  Overdue  Assignment,  Jen- 
nifer Lewington  and  Graham  Orpwood 
use  some  superb  examples  and  analogies 
in  the  chapter  called  “Measuring  Suc- 
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cess”  to  explain  terminology  and  to  clear 
up  misunderstandings. 

The  fourth  task  was  to  differentiate 
between  “standards,”  “standardization,” 
and  “standardized  tests.”  Over  the  past 
ten  months,  the  major  media  have  grap- 
pled with  the  notion  of  “standards  versus 
standardized.”  The  issue  has  been  further 
complicated  by  the  fact  that  the  term 
“standardized,”  as  it  refers  to  testing,  has 
been  broadened  by  researchers  in  some 
circles  to  mean  any  large-scale  test. 
Teachers  tend  to  think  of  standardized 
tests  as  norm-referenced,  paper  and  pen- 
cil, group,  multiple-choice  tests.  Teach- 
ers question  their  validity  and  their 
relevance,  and  Ontario  students  don’t  get 
much  practice  doing  them.  Lack  of  skill 
in  doing  these  tests  could  be  one  of  the 
reasons  why  Ontario  students  have  not 
performed  well  on  such  tests.  Parents 
and  others  tend  to  refer  to  any  test  or 
exam  that  a large  number  of  students 
participate  in  as  a standardized  test  and 
some  poll  results  suggest  that  parents 
want  more  “standardized  tests.”  In  the 
Benchmarks  Project,  we  are  talking 
about  standards ! In  fact,  we  are  talking 
about  high,  worthwhile,  and  consistent 
standards  for  all,  even  if  some  (or  many) 
cannot  yet  reach  them  (Wiggins,  1991). 
This  has  been  a very  difficult  shift  for 
people  to  make. 

The  reasons  for  the  difficulty  are 
obvious  and  they  lead  to  the  next  shift  in 
thinking.  What  is  a standard  and  how  are 
we  defining  a standard?  We  have  used 
the  work  of  Grant  Wiggins  (cited  above) 
to  define  a standard  and  the  work  of  D. 
R.  Sadler  (1987)  in  the  Oxford  Review  of 
Education  as  the  basis  for  determining 
how  standards  would  be  set.  What  we 
learned  as  we  worked  with  Wiggins’s 
notion  across  the  province  was  that  very 
few  people  could  conceive  of  a standard 
as  anything  but  a number  or  a rank. 
Sadler  identifies  five  ways  of  setting 
standards  and  we  have  tried  to  give  some 
examples  of  each  in  Figure  2.  Once  we 
had  done  the  research  and  identified  the 
outcomes  expected  by  futurists,  profes- 
sionals, and  teachers,  we  developed  the 
initial  provincial  benchmarks  or  stan- 
dards by  eliciting  the  tacit  knowledge 
from  expert  teachers  (type  3).  These 
standards  were  then  written  down  (verbal 
statements,  type  4)  and  indicators  were 
added  to  make  them  more  explicit.  Often 
to  determine  the  indicators  or  evidence 
(criteria,  type  2),  we  had  to  go  to  key 


Figure  2 Five  Known  Methods  of  Devel- 
oping Standards 


1 A number,  a norm,  often  an  average.  This 
number  is  used  as  a standard  against 
which  to  measure  performance  over  time, 
or  as  a basis  for  ranking  (e.g.,  I.Q.  test 
scores). 

2 A criterion.  A number  reference  or  set 
of  criteria  against  which  performance  is 
measured  (e.g.,  words  per  minute  in 
typing). 

3 Tacit  knowledge.  The  knowledge  of  the 
expert  or  connoisseur.  These  standards 
are  resident  in  the  head  of  the  expert,  but 
are  not  available  to  others  (e.g,  Olympic 
diving). 

4 Verbal  description  A statement  makes  the 
standard  explicit  public  and  accessible 
so  the  standard  can  be  known,  met  chal- 
lenged (e.g.,  job  descriptions). 

5 Key  examples.  Examples  of  the 
performance  or  behaviour  are  shown 
(visual  or  text)  to  help  people  recognize 
the  standard  when  they  see  it  They  are 
not  explicit  standards,  they  represent 
standards  (e.g,  videos  of  performance). 


examples  of  student  work  (type  5)  or  to 
norm-referenced  test  results  (type  1).  It 
was  our  intent  to  embrace  all  the  differ- 
ent types  of  standards  in  the  project  so 
that  it  would  not  be  viewed  as  narrow  or 
only  serving  certain  purposes.  Sadler’s 
work  provided  a way  for  us  to  be  consis- 
tent with  our  own  principles. 


The  next  shift  in  thinking  was  to  have 
teachers,  parents,  and  the  public  come  to 
a common  understanding  about  the  terms 
test  and  assessment.  Again,  the  chapter 
on  accountability  in  Overdue  Assignment 
provides  clarity  about  the  appropriate 
uses  and  the  differences.  A test,  as  the 
word  is  traditionally  used,  measures  per- 
formance under  controlled  conditions  at 
one  point  in  time.  Assessment  implies  a 
broad  range  of  instruments  and  methods 
used  to  used  to  ensure  that  the  desired 
standard  has  been  met  and  that  the 
achievement  or  performance  is  consis- 
tent over  time.  Performance  assessment 
usually  refers  to  the  measurement  of 
complex  rather  than  simple  tasks. 

Finally,  in  the  framework,  the  shift  in 
the  way  we  think  about  reporting  had  to 
be  dealt  with.  There  is  a great  deal  of 
controversy  about  how  results  should  be 
reported.  There  is  not  a clear  differentia- 
tion in  the  mind  of  educators  or  the  public 
of  different  types  of  accountability  — fis- 
cal accountability,  performance  manage- 
ment of  staff,  or  quality  of  student  results. 
Information  about  context  must  be  avail- 
able before  decisions  or  judgments  can  be 
made  on  the  basis  of  data.  If  we  accept 
the  principles  on  which  the  project  is 
based,  then  data  can  only  be  reported  for 
purposes  of  improvement  and  only 
against  the  standards.  (See  Figure  3.) 

The  data  are  not  relevant  unless  they 
are  in  context.  Given  the  current  world 


Figure  3 From  Data  to  Decisions  and  Actions 


TYPES 


STUDENT  PROGRAM 

STEPS  (Teacher  Level)  (School,  Board,  Provincial  Level) 


Assessment 

Collection  of  a wide 
range  of  data  about  an 
individual  student's 
achievement 

Collection  of  a wide  range  of  data  about 
overall  achievement  in  the  jurisdiction 

Evaluation 

Reporting  to 

• student 

• parent 
about  individual 
achievement  as 
referenced  to  a 
standard 

Reporting  to 

• teachers 

• principals 

• boards 

• public 

about  achievement  of  the  jurisdiction  as 
referenced  to  a standard 

Decisions 

• remediation 

• enrichment 

• placement 

• promotion 

• change  policy 

• provide  staff  development 

• realign  funding  i 
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and  economic  context  described  in  the 
opening  paragraph  of  this  article,  it  will 
be  very  hard  to  maintain  these  principles 
at  the  provincial,  school  board,  or  school 
level.  If  we  don’t,  however,  we  will  end 
up  spending  a great  deal  of  money  testing 
against  standards  where  the  data,  when 
reported,  tell  us  very  little  about  actual 
achievement.  This  will  happen  because 
schools  will  focus  on  ensuring  a high 
rank  in  the  reporting  of  results  instead  of 
focussing  on  improving  programs  at  the 
individual  child,  school,  or  system  level. 

In  the  end,  if  the  principles  and  frame- 
work are  to  result  in  improved  student 
achievement,  then  we  must  move  from 
outcomes  to  standards,  from  assessment 


to  data,  to  decisions,  and  to  action. 

This  description  of  the  development 
process  is  a retrospective  one.  The  pro- 
cess as  it  evolved  was  messy,  slow,  often 
two  steps  forward  and  one  back.  We 
were  breaking  new  ground,  working  at 
tasks  and  in  ways  that  we  believe  are 
consistent  with  education  systems 
around  the  world  as  they  struggle  to  pre- 
pare their  students  (and  their  teachers) 
for  the  21st  century.  The  real  test  will  be 
the  product;  the  draft  provincial  stan- 
dards which  have  been  developed  are 
described  in  two  other  articles  in  this 
issue.  They  will  be  available  to  schools 
and  across  the  province  in  the  fall  of 
1993  as  interim  standards  in  Grades  3,  6, 


and  9 mathematics  and  language.  They 
will  be  reviewed  by  teachers,  parents, 
and  the  public,  and  ultimately  it  will  be 
the  Royal  Commission  on  Learning  that 
recommends  the  accountability  measures 
that  will  remain  or  move  into  place  in 
Ontario  in  the  future. 
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From  Curriculum 
to  Outcomes-Based 
Education 

Provincial  Language  Standards 


Neil  Graham 

Learning  Assessment  Branch, 

Ontario  Ministry  of  Education 
and  Training 

Throughout  the  English-speaking  world, 
the  last  ten  years  have  seen  a dramatic 
shift  from  curriculum-based  education  to 
outcomcs-based  education.  In  Ontario, 
this  move  has  had  a powerful  impact  on 
education  as  teachers  and  administrators 
at  all  levels  attempt  to  develop  standards 
and  assessment  methods  which  focus  on 
outcomes  and  at  the  same  time  acknowl- 
edge the  complexity  of  learning  and  of 
assessment.  Those  who  were  involved  in 
any  way  in  the  1991/92  Provincial 
Review  of  Grade  12  Writing  will  be  well 
aware  of  this  complexity.  Designing  and 
administering  the  assessment  activity, 


assessing  students’  writing,  and  setting 
the  provincial  standards  against  which  to 
judge  their  writing  performance  all 
involved  complex  but  necessary  pro- 
cedures that  ensured  that  the  link  between 
learning  and  assessment  was  maintained. 

Over  the  past  three  years,  the  Ministry 
of  Education  and  Training  has  been 
working  on  this  shift  in  emphasis  from 
curriculum  to  outcomes-bascd  education 
on  two  fronts:  developing  provincial 
standards  (benchmarks)  in  mathematics 
and  language,  and  developing  a Com- 
mon Curriculum  which  reorganizes  cur- 
riculum delivery  for  Grades  3,  6,  and  9 
and  articulates  specific  outcomes  for  the 
end  of  these  divisions. 

Because  the  development  of  mathe- 
matics standards  began  a year  earlier  than 
language  standards,  they  arc  at  different 
stages  of  development.  Language  stan- 


dards have  been  under  development  for 
the  past  year  and  have  just  now  reached 
the  stage  where  draft  material  can  be  pre- 
sented to  teachers,  administrators,  and  the 
public  for  response.  Over  the  next  18 
months,  both  the  mathematics  and  lan- 
guage standards  will  be  treated  as  interim 
standards,  subject  to  modification  and 
refinement.  The  draft  language  bench- 
marks (see  Figure  3)  will  be  available  in 
schools  in  the  fall.  In  the  meantime,  this 
article  provides  an  overview  of  the  devel- 
opment to  date  and  a first-draft  sample. 

Creating  a Framework  for 
Provincial  Language  Standards 

During  the  past  year,  the  Ministry  of 
Education  and  Training  has  had  two 
committees  working  on  the  development 
of  provincial  language  standards  for 
Grades  3,  6,  and  9.  A steering  committee 
representing  a variety  of  educational 
groups  has  provided  leadership  and 
direction  in  the  articulation  of  guidelines 
upon  which  to  base  the  development  of 
the  standards,  and  a writing  committee  of 
classroom  teachers  and  consultants  has 
developed  the  provincial  language  stan- 
dards from  the  guidelines  and  framework 
established  by  the  steering  committee. 

The  following  material  provides  an 
overview  of  the  conceptual  framework 
and  some  sample  material  the  commit- 
tees have  developed  for  establishing  and 
using  provincial  language  standards  to 
assess  student  performance. 
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Figure  1 The  Link  Between  Outcomes,  Standards,  Assessment,  and  Reporting 


DETERMINING 

OUTCOMES 

Outcomes  state  general 
results  expected. 

• The  Common  Ctrriculum 
(Grades  1-9)  determines  the 
knowledge,  skills  and  values 
expected  for  all. 

SETTING 

STANDARDS 

Standard  state  the  expected 
range  of  student  achievement 
for  the  outcomes. 

• The  province  will  provide  clear 
standards  in  language  and 
mathematics. 

ASSESSING 

PERFORMANCE 

Performance  will  be  assessed 
against  the  provincial 
standards  in  language  and 
math  (Grades  3, 6,  and  9). 

• Classroom  teachers  will  have 
common  provincial  standards 
as  a basis  for  evaluating  and 
reporting  student  achievement 

• School  boards  and  the  province 
will  use  the  same  standards  for 
board-wide  and  provincial  testing 

REPORTING 

RESULTS 

Results  will  be  reported  in 
relation  to  the  provincial 
standards  in  language  and 
math  (Grades  3, 6,  and  9). 

• Classroom  teachers  will,  where 
appropriate,  report  student 
achievement  in  relation  to  the 
standards. 

• School  boards  and  the  province 
will  report  school-wide,  school 
board  and  provincial 
performance  in  relation  to  the 
standards. 

Guidelines  for  Creating 
Provincial  Language  Standards 
for  Grades  3,  6,  and  9 

After  considerable  deliberation  and 
review  of  existing  approaches  to  stan- 
dards-setting  from  various  provincial, 
national,  and  international  jurisdictions, 
the  steering  committee  agreed  on  the  fol- 
lowing guidelines  for  developing  the 
provincial  language  standards  for  the 
Province  of  Ontario: 

1.  The  language  standards  would  be 
based  on  the  language  outcomes 
described  for  Grades  3,  6,  and  9 in  the 
Common  Curriculum.  These  out- 
comes are  described  in  terms  of 
knowledge,  skills,  and  values  that  are 
consistent  with  the  developmental 
stages  of  language  acquisition. 

2.  One  set  of  language  standards  would 
be  developed  for  all  students,  includ- 
ing ESL/ESD  students  and  special 
education  students.  The  standards 
would  accommodate  the  present 
Ontario  practice  of  age- appropriate 
placement  of  these  students. 

3.  Although  reading,  writing,  and  speak- 
ing and  listening  activities  are  inte- 
grated and  should  be  assessed  in  an 
integrated  manner,  for  purposes  of 
assessing  performance,  separate 
provincial  language  standards  would 
be  developed  for  each  of  reading, 
writing,  and  speaking  and  listening. 

4.  For  each  of  Grades  3,  6,  and  9,  the 
provincial  language  standard  for  each 
language  area  would  be  described  in 
terms  of  an  expected  range  of  perfor- 
mance, in  effect  establishing  an  all- 
inclusive  provincial  performance  scale 
or  a scale  for  each  language  area  and 
each  identified  grade  level.  The  scale 
would  describe  six  levels  of  perfor- 
mance, with  the  standard  — that  is, 
the  expected  range  of  performance  for 
that  grade  — being  levels  three,  four, 
and  five.  Performance  at  levels  one 
and  two  would  be  considered  to  not 
yet  be  at  the  expected  performance, 
and  level  six  performance  to  be 
beyond  the  range  of  expected  perfor- 
mance for  that  grade.  Specific  assess- 
ment procedures  would  require  the 
development  of  specific  scales  devel- 
oped from  pertinent  information  found 
in  the  scales  which  describe  the  total 
performance.  (See  Figures  1 and  2.) 


5.  The  scales  would  demonstrate  grada- 
tions in  performance  within  the  six 
levels  for  each  of  the  three  identified 
grades  and  demonstrate  horizontal 
coherence  in  expected  performance 
across  the  three  grades.  However,  the 
scales  across  the  grade  level  should 
not  be  developed  in  an  overlapping 
manner;  for  example,  level  five  in 
Grade  3 should  not  be  seen  as  a level 
two  performance  in  Grade  6. 

Developing  the  Provincial 
Language  Standards 
For  Grades  3, 6,  and  9 

Using  the  various  methods  by  which 
standards  are  developed  (see  Gail  Rap- 
polt’s  article  for  a description  of  those 
methods),  the  writing  committee  has 
articulated  six  levels  of  performance  for 
each  language  area  as  follows. 

Each  level  is  identified  by  a word  that 
describes  the  quality  of  the  overall  per- 
formance. This  is  followed  by  a general 
descriptor  indicating  what  the  overall 
performance  in  the  specific  language 


Figure  2 Provincial  Language 
Standards  Framework 


LEVEL  6 

Superior 

Performance 

Students  performing 
beyond  expected 
range  of  standards 
set  for  grade. 

LEVEL  5 

Proficient 

Performance 

PROVINCIAL 

STANDARDS 

Students 
performing  within 
the  expected  range 
set  for  grade. 

LEVEL  4 

Capable 

Performance 

LEVEL  3 

Adequate 

Performance 

LEVEL  2 

Limited 

Performance 

Students  not  yet 
performing  within 
expected  range 
set  for  grade 

LEVEL  1 

Rudimentary 

Performance 
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Figure  3 Draft  Language  Standards  (draft  June  4, 1993) 


PROVINCIAL  STANDARDS  - GRADE  9 WRITING  AND  REPRESENTING 

Level  6 - Superior  Writing  Performance 

The  student  is  an  independent  and  self-confident  writer  who  has  internalized  and  adapted  the  various  stages  of  the  writing  process  to  the 
point  where  they  are  applied  automatically.  The  writing  exhibits  a strong  voice  and  a command  of  a wide  range  of  forms,  genres,  language 
conventions,  and  stylistic  devices.  The  writing  is  thoughtful,  clear,  imaginative,  and  often  innovative. 


Indicators 

Characteristics: 


The  Process: 


Attitude: 


The  content  is  compelling.  The  focus  is  complex  and  is  supported  by  a wealth  of  original  and  perspicuous  detail. 

The  organization  is  expert.  There  is  some  surprising  subtlety  in  the  arrangement. 

The  writing  evidences  an  impressive  command  of  the  resources  of  language:  a wide  vocabulary,  a full  range  of  sentence 
patterns,  control  over  stylistic  devices. 

The  writing  evidence  complete  control  over  the  conventions  of  the  form  in  which  it  is  operating. 

The  writer  monitors  his  or  her  own  growth  as  a writer;  sets  long  range  goals  for  improvement  and  tracks  own  progress  „ 
toward  those  goals. 

The  writer  can  access  all  the  features  of  desk  top  publishing  and  graphics  programs  to  integrate  art,  graphics  and  text  in 
a polished  piece  of  work. 

Strongly  values  and  enjoys  writing.  Appreciates  the  full  range  of  purposes  of  writing;  to  discover,  shape  and 
communicate  personal  meaning;  to  enter  into  and  contribute  to  discourse  communities;  offer  to  access  information  and 
ideas  across  time  and  space. 


Level  5 * Proficient  Writing  Performance 

The  student  Is  an  Independent  and  self-confident  writer  who  ha9  largely  internalized  the  various  stages  of  the  wrSIng  process.  The 
writing  exhibits  a clear  voice  and  a command  of  a range  of  forms  and  genres,  language  conventions  and  some  stylistic  devices.  The 
writing  is  thoughtful,  clear,  and  imaginative. 


Indicators 

Characteristics: 


The  Process: 


Attitude: 


» Has  a fairly  complex  focus  and  a wealth  of  relevant  and  convincing  supporting  detail. 

• Has  a complex  hierarchical  organization  that  is  sustained  throughout 

• Is  a written  with  full  considerable  command  of  the  resources  the  language;  It  can  vary  the  tone,  mood,  style  to  suit 
the  purpose- 

• Has  a fine  grained  appreciation  of  the  characteristics  of  the  genre  on  which  I Is  operating, 

• Understands  the  writing  process  to  Include  long  range  planning  as  weS  as  a recursive  approach  to  individual  pieces. 

• Understands  and  can  draw  on  a variety  of  strategies  tor  generating  and  improving  text 

• Can  access  mechanical  style  and  grammar  analysis  programs 

• Values  and  enjoys  writing  appreciates  role  on  shaping  personal  and  cultural  identity;  enjoys  and  contributes  to 
community  of  writers  In  the  classroom. 


Level  4 - Capable  Writing  Performance 

The  student  Is  largely  an  independent  and  self-confident  writer,  depending  on  the  writing  task,  and  is  generally  skilful  in  applying  the 
stages  of  the  writing  process.  The  writing  exhibits  some  sense  of  voice  and  a control  of  a range  of  forms,  genres,  language  conventions, 
and  some  stylistic  devices  are  present.  The  writing  Is  generally  thoughtful,  clear,  and,  at  times,  imaginative. 


Indicators 

Characteristics: 


The  Process. 


Attitude: 


The  content  has  clear  focus  with  relevant  supporting  detail. 

The  material  is  discernably  organized  in  an  orderly  progression  of  sections  with  explicit  transitions. 

It  is  written  In  correct  and  continuous  prose  that  complies  with  the  conventions  ot  spelling,  punctuation  and  grammar. 
It  demonstrates  a familiarity  with  the  conventions  of  the  form  or  genre  in  which  It  Is  operating. 

Understands  the  processes  of  planning,  drafting,  revising. 

Seeks  out  advice  and  feedback  at  alt  stages  of  the  process. 

Can  access  regular  features  ol  word  processing. 

Values  the  place  of  writing  In  school  and  society. 
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Figure  3 Continued 


Level  3 * Adequate  Writing  Performance 

The  student  has  some  self-confidence  as  a writer  and  is  generally  skilful  in  applying  most  of  the  various  stages  of  the  writing  process. 
The  writing  exhfoits  a sense  of  voice  at  times  and  a control  of  basic  forms,  genres,  and  language  conventions,  and  some  attempt  at 
using  stylistic  devices  is  evident.  The  writing  is  generally  clear. 

Indicators 


Characteristics: 


The  Process: 


Has  a sustained  organization  throughout,  using  Simple  ordering  and  listing  arrangements. 

Has  discardable  form  and  is  working  within  the  recognizable  characteristic  conventions  of  a genre. 

Is  written  in  continuous  prose. 

The  writer  exercises  considerable  control  over  vocabulary,  spelling,  and  conventional  punctuation. 

Has  an  organized  system  for  storing  work  in  progress  and  saving  completed  work. 

Understands  revising  to  include  improving  substantive  content  and  organization  as  well  as  surface  features. 
Regularly  revises  working  in  progress. 


Attitude: 


Is  beginning  to  appreciate  writing  as  a learning  tool. 


Level  2 - Limited  Writing  Performance 

The  student  is  a developing  writer  who  has  some  self-confidence  with  certain  aspects  of  writing  and  is  able  to  apply  some  of  the  stages  of 
the  writing  process  with  guidance.  The  writing  exhibits  capability  with  some  of  the  basic  forms,  genres,  and  language  conventions.  The 
writing  demonstrates  clarity  at  times. 

Indicators 

Characteristics:  • 


Attitude : 


There  is  an  attempt  at  organization:  effective  opening  and  closing  sections  and  some  attempts  to  order  the  material  in 
between  around  a main  idea. 

Words  are  strung  together  in  intelligible  phrases  and  sentences  but  the  conventions  of  spelling,  marking  sentence 
boundaries  are  not  evident. 

There  is  some  attempt  at  form,  but  only  the  broadest  characteristics  of  the  form  are  evident. 

The  writer  revises  work  independently  or  collaboratively  to  correct  surface  features:  spelling  and  punctuation. 

The  writer  is  beginning  to  gain  personal  satisfaction  from  writing. 


Level  1 - Rudimentary  Writing  Performance 

The  student  is  an  emerging  writer  who  can  apply  some  of  the  stages  of  the  writing  process  with  guidance.  The  writing  exhibits  a 
rudimentary  grasp  of  some  of  the  basic  forms,  genres,  and  application  of  a few  of  the  basic  language  conventions. 

Indicators 


Characteristics: 


Sentence  fragments 

Minimal  information  with  shifting  focus. 

Organization  limited  to  primitive  opening  and  closing  statements. 

Has  limited  words  and  structures  available  for  expressing  meaning;  controls  everyday  vocabulary  words  but  has  difficulty 
stringing  them  together. 

Has  a limited  and  undifferentiated  sense  of  form  or  genre  and  audience. 

Prefers  oral  to  written  expression. 

Unconventional  spelling  of  frequently  used  vocabulary. 


The  Process: 


Will  revise  work  with  support,  direction,  and  encouragement. 
Revises  by  recopying  to  improve  legibility. 


Attitude: 


Needs  to  be  encouraged  and  motivated  to  write. 
Prefers  writing  that  relates  to  personal  experience. 
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area  looks  like,  and  that  descriptor  is  fol- 
lowed by  a set  of  indicators  providing 
detailed  description  of  various  elements 
of  the  performance.  The  general  scale 
describing  provincial  standards  for 
Grade  9 Writing  (see  Figure  3)  demon- 
strates this  pattern.  (Keep  in  mind  that  at 
this  stage,  the  scales  are  written  in  pro- 
fessional language  for  use  by  teachers.) 

Using  the  Provincial 
Language  Standards 

Teachers  and  jurisdictions  wishing  to 
assess  student  performance  against 
provincial  standards  will  use  these  gener- 
ic performance  descriptors  and  select 
and/or  adapt  the  indicators  that  apply.  For 
example,  if  their  assessment  activity  were 
broad  enough  to  include  all  the  indicators 
identified  in  any  one  language  area  — for 


instance,  assessing  student  writing 
through  writing  folder  assessment  — they 
would  be  able  to  use  the  general  scale  as 
is.  If,  however,  the  assessment  activity 
were  narrower  — a single  piece  of  writ- 
ing or  reading  passage  or  an  oral  presen- 
tation — the  descriptors  and/or  indicators 
would  need  to  be  narrowed  to  what  is 
being  measured  by  the  assessment. 

The  Ministry  will  use  the  general 
provincial  scales  and  the  identified 
provincial  standards  for  reading  and  for 
writing  to  assess  students’  level  of  per- 
formance on  the  Grade  9 Reading  and 
Writing  test  this  fall.  Since  the  test  asks 
students  to  respond  to  a variety  of  read- 
ing passages  and  to  write  a variety  of 
compositions  over  a period  of  time,  the 
general  descriptors  and  most  of  the 
indictors  will  apply. 


Follow-up  and  Response 

The  committees  will  complete  work  on 
these  provincial  standards  over  the  sum- 
mer and  in  the  early  fall  schools  will 
receive  a complete  set  of  draft  materials 
for  reading,  writing,  and  speaking  and 
listening  for  Grades  3,  6,  and  9.  Teachers 
and  interested  groups  are  asked  to  exam- 
ine these  interim  standards  over  the  next 
18  months  and  respond  to  them. 


Anyone  wishing  to  respond  to  the 
overview  material  presented  in  this  arti- 
cle is  asked  to  contact  the  Learning 
Assessment  Branch,  Ministry  of  Educa- 
tion and  Training,  18th  Floor  Mowat 
Block,  900  Bay  Street,  Toronto  M7A 
1L2;  (416)  327-9071. 


Improving  Student 
Results  in  Mathematics 

Developing  Standards  of  Achievement 


Gerald  Hurtubise 

Eastern  Ontario  Regional  Office, 
Ontario  Ministry  of  Education 
and  Training 


In  response  to  public  pressure  to  deliver 
improved  student  results  in  mathematics, 
the  Minister  of  Education  requested,  in 
the  fall  of  1991,  that  provincial  standards 
for  mathematics  (for  Grades  3,  6,  and  9) 
be  the  first  priority  of  the  Provincial 
Benchmarks  Project.  In  February,  1992, 
increased  impetus  for  the  math  project 
came  with  the  release  of  the  results  from 
the  International  Assessment  of  Educa- 
tional Progress  and  the  Provincial 
Review  results  for  Grades  8,  10,  and  12. 

The  purpose  of  developing  mathemat- 
ics benchmarks  is  the  improvement  of 
student  results  in  mathematics.  Standards 
will  be  used  to  identify  areas  where 
instruction  can  be  improved,  to  diagnose 


shortcomings,  to  determine  necessary 
improvements,  and  to  provide  means  for 
the  enrichment  and  enhancement  of 
mathematics  programs  in  both  English- 
and  French-language  schools  across  the 
province.  They  are  to  be  standards  for 
student  achievement  and  assessment  as 
well  as  standards  for  the  monitoring  of 
programs. 

The  development  and  production  of 
these  standards  arc  the  result  of  close 
collaboration  among  mathematics  teach- 
ers, Ministry  of  Education  and  Training 
officials,  and  representatives  of  the  three 
mathematics  associations  (two  English 
and  one  French)  in  Ontario.1 

In  researching  and  developing  the 
mathematics  standards,  the  research  team 
looked  at  curriculum  and  assessment 
materials  from  school  boards  in  Ontario 
and  other  provinces  in  Canada.  Work 
from  oilier  English  and  French  speaking 
countries  was  carefully  analysed.  Where 
possible,  translations  and  summaries  of 


directions  in  Asian  and  other  European 
countries  were  reviewed.  It  is  important 
to  note,  however,  that  although  material 
from  other  countries,  particularly  those 
whose  values  and  school  culture  are  radi- 
cally different  from  Ontario’s,  can  yield 
valuable  information,  it  cannot  provide 
simple  recipes  to  follow. 

The  document  “Provincial  Mathemat- 
ics Benchmarks”  provides  a framework 
for  standards  in  mathematics  for  Grades 
3,  6,  and  9.  The  material  in  the  document 
is  organized  in  two  ways:  by  grade  and 
by  components  of  mathematics  as 
defined  by  Ontario  mathematics  educa- 
tors. These  components  have  been 
grouped  into  the  following  six  strands  in 
the  document: 

1 Inquiry  and  Problem  Solving 

2 Number  Sense  and  Numeration 

3 Spatial  Sense  and  Geometry 

4 Patterning  and  Algebra 

5 Data  Management  and  Probability 

6 Estimation  and  Measurement 

For  each  strand,  for  each  grade,  there 
arc  one  or  two  statements  of  expected 
learning  outcomes  (tied  to  the  outcomes 
in  die  Common  Curriculum),  statements 
of  the  standards  or  benchmarks,  sample 
evidence  to  clarify  these  statements,  and 
sample  assessment  activities  that  give 
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key  examples  to  indicate  whether  the 
standard  had  been  met.  (See  Figure  1; 
sample  assessment  activities  have  not 
been  included  in  this  article.) 

The  benchmark  statements  them- 
selves indicate  a provincial  standard  for 
all  students,  followed  by  additional 
achievement  for  most  students.  A further 
level  of  achievement,  which  some  stu- 
dents will  attain  by  the  end  of  the  grade, 
is  also  described.  This  three-part  struc- 
ture is  intended  to  accommodate  the 
views  of  teachers  and  community  groups 
who  recommended  that  the  provincial 
standard  of  achievement  should  not  be 


set  impossibly  high,  but  that  neither 
should  only  a minimum  level  of  achieve- 
ment be  accepted. 

The  standards  document  does  not  dis- 
cuss or  recommend  methods  of  instruc- 
tion to  be  used  in  the  classroom.  Howev- 
er, the  performances  described  in  the 
benchmarks  indirectly  indicate  the  expe- 
riences or  practice  students  need  to  have 
in  order  to  meet  the  standards.  Figure  2 
outlines  the  fundamental  principles  of 
mathematics  education  presented  by  the 
Ontario  Mathematics  Co-ordinators 
Association.  The  principles  are  consis- 
tent with  the  beliefs  and  teaching  stan- 


Figure  2 Fundamental  Principles  of 
Mathematics  Education  Grades  1 to  9 


Throughout  the  process  of  instruction  and 

student  assessment  for  mathematics,  it  is 

assumed  that: 

• all  learning  will  be  within  a 
developmental  framework 

• technology,  problem  solving  estimation 
strategies  will  be  integrated  throughout 

• students  will  be  working  with  a wide 
variety  of  manipulative  materials 

• students  will  be  looking  for  patterns  on 
which  to  make  generalizations  and 
predictions 

• the  teaching  strategies  are  activity 
based,  collaborative  and  co-operative 

• the  learning  tasks  are  authentic  and 
integrated 

• conclusions  will  be  justified  verbally  or 
in  writing 


dards  expressed  by  the  National  Council 
of  Teachers  of  Mathematics  in  their  stan- 
dards documents.  (They  also  seem  con- 
sistent with  the  views  expressed  by  Ken 
Travers  of  the  National  Science  Founda- 
tion in  this  issue  of  Orbit. ) 

The  standards  also  reflect  the  mathe- 
matics component  of  the  learning  out- 
comes embedded  in  the  Common  Cur- 
riculum in  the  Math,  Science,  and  Tech- 
nology core  program  area.  The  learning 
outcomes  in  the  Common  Curriculum 
describe  achievement  and  performance 
that  cross  the  traditional  lines  of  sub- 
jects; the  benchmarks  ensure  that  the 
basic  knowledge,  skills,  and  beliefs  nec- 
essary for  competence  in  mathematics 
are  achieved. 

The  process  of  setting  publicly  agreed 
upon  standards  is  far  from  complete.  The 
draft  document  was  developed  to  provide 
a basis  for  discussion  in  order  to  refine 
and  validate  the  standards  it  contains.  It 
was  circulated  for  comment  as  widely  as 
possible  during  the  fall  of  1992,  accom- 
panied by  a validation  response  form. 
This  form  asked  general  questions  about 
the  framework,  structure,  and  approach 
of  the  document  and  also  asked  for  very 
specific  comment  on  the  levels  of  stu- 
dent achievement  expected  in  each  of  the 
standards. 

During  the  winter  of  1992/93  over 
525  responses  to  the  document  were 
reviewed.  The  results  of  the  analysis  are 
now  being  turned  over  to  a team  of 
teachers  who  will  rewrite  the  standards 
based  on  public  input.  In  very  broad 


Figure  1 Primary  Division,  Strand  2 (draft) 


CRITICAL  LEARNING  OUTCOMES 

Understand  the  fundamental  operations  of  addition,  subtraction,  multiplication 
and  division  and  their  inter-relationships. 

BENCHMARKS 

SAMPLE  EVIDENCE 

All  students  will: 

given  a real-life  context  and 
appropriate  resources, 
demonstrate  the  fundamental 
operations  accurately. 

— demonstrate  an  understanding  of  addition  and 
subtraction  with  concrete  materials. 

— know  basic  facts  without  the  use  of  concrete  materials 
or  calculators  as  defined  below: 

Addition:  up  to  10  + 10 

Subtraction:  up  to  20  - 10 

Multiplication:  up  to  5 x 5.  i 

— demonstrate  the  addition  and  subtraction  of  two  2-  and 
3-digit  numbers  using  concrete  materials  (with  and 
without  regrouping). 

— demonstrate  understanding  of  multiplication  and  its 

relationship  to  addition  using  concrete  materials.  : 

— using  concrete  materials  or  representations,  1 

demonstrate  multiplication  and  division. 

— identify  the  operation  required  to  solve  a simple  problem. 

In  addition,  most  students  will: 

given  a real-life  context  and 
appropriate  resources,  perform 
the  fundamental  operations 
accurately  and  explain. 

— demonstrate  the  addition  and  subtraction  of  2- and 
3-digit  numbers  using  paper  and  pencil  procedures 
(with  and  without  regrouping). 

— demonstrate  an  understanding  of  division  and  its 
relationship  to  subtraction  using  concrete  materials. 

— know  basic  facts,  without  the  use  of  concrete  materials 
or  calculators,  as  defined  below: 

Division:  up  to  25 + 5. 

— identify  and  use  the  specific  operations  of  addition  and 
subtraction  in  a problem  solving  situation. 

— create  and  solve  problems  using  the  operations  of 
addition  and  subtraction. 

— use  the  relationship  between  related  operations  to 
verify  results. 

In  addition,  some  students  will: 
given  a real-life  context,  perform 
the  fundamental  operations 
accurately  without  using 
resources,  and  explain  and 
demonstrate  the  computation. 

— know  basic  facts  as  defined  below: 

Multiplication:  up  to  10  x 5 

Division:  up  to  50  + 5. 

— create  and  solve  problems  which  demonstrate  the 
operations  (+,-,x,+)  in  a given  situation 

— add  and  subtract  two  numbers  with  more  than  3 digits 
(with  or  without  regrouping). 
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terms,  the  results  were  as  follows: 

• In  general,  there  was  very  strong  sup- 
port for  the  development  of  clear  and 
publicly  agreed  upon  standards. 

• There  was,  however,  a general  lack  of 
understanding  and  agreement  about 
what  a standard  actually  is  and  for 
what  purposes  it  should  be  used. 

• Although  there  was  general  agree- 
ment that  the  standards  should  identi- 
fy a range  of  performance,  rather  than 
a single  level,  the  use  of  the  terms  “all 
students,”  “most  students,”  and  “some 
students”  was  generally  described  as 
unacceptable. 

• There  were  opposing  opinions  ex- 
pressed about  the  appropriate  weight 
given  to  geometry,  data  management, 
patterning  and  algebra,  and  complex 
problem  solving  in  relation  to  “the 
basics”  of  numeracy  and  measurement. 

• Many  educators  expressed  the  view 
that  the  standards,  as  written,  were  not 
currently  attainable  by  our  students; 
those  outside  education  tended  to  rate 
the  standards  as  too  low  overall. 

There  is  also  a problem  of  language. 


How  can  mathematics  educators  use  the 
language  of  mathematics  to  state  high 
and  consistent  standards  in  the  discipline 
and  still  meet  the  need  for  plain,  clearly 
understandable  language?  That  is  a 
dilemma  yet  to  be  resolved. 

During  the  summer  of  1993,  revised 
standards  that  take  all  the  concerns  and 
diverse  opinions  intq  consideration  will 
be  prepared.  Some  of  the  changes  will  be 
straightforward;  some  will  require  judg- 
ment and  therefore  not  please  everyone. 
The  current  work  being  done  internation- 
ally on  mathematics  performance  stan- 
dards and  the  work  that  has  been  done 
over  the  last  year  in  developing  the  lan- 
guage standards  for  Ontario  have  provid- 
ed new  insights  and  have  led  to  a slightly 
revised  structure  for  the  mathematics 
standards. 

The  Royal  Commission  on  Learning, 
formed  in  early  May  1993,  will  review 
the  broad  issues  of  educational  account- 
ability over  the  next  18  months.  The  draft 
provincial  mathematics  standards  will  be 
interim  standards  during  the  term  of  the 
Royal  Commission  and  will  be  available 
in  the  schools  in  early  fall,  1993.  They 
will  provide  standards  against  which  to 
measure  individual  student’s  performance 


and  a basis  for  discussion  between  par- 
ents and  teachers.  As  well,  they  will  be 
the  basis  for  discussion  in  the  broader 
community  — discussion  that  will  have 
to  answer  two  questions: 

“Do  these  standards  measure  the 
important  mathematical  outcomes  we 
want  students  to  learn?” 

“Are  these  mathematics  standards 
high  enough?” 

If  you  have  comments  or  questions 
about  the  Ontario  Mathematics  Standards 
or  if  you  would  like  a copy  of  the  draft 
document  in  French  or  in  English,  please 
contact  Gerald  Hurtubise,  Eastern  Ontario 
Regional  Office,  Ministry  of  Education 
and  Training,  4th  Floor,  1580  Merivale 
Road,  Nepean,  Ontario,  K2G  4B5;  (613) 
225-9210,  extension  190. 


NOTE 

1.  The  co-operation  and  assistance  of  the  teachers, 
researchers,  mathematics  co-ordinators,  and  mem- 
bers of  the  Ontario  Association  for  Mathematics 
Education  who  contributed  to  the  development  of 
the  mathematics  standards  have  been  very  much 
appreciated  by  Ministry  staff.  Also  TVO’s  Decem- 
ber 4,  1992,  program  “Responding  to  the  Provincial 
Mathematics  Benchmarks”  has  been  a useful  staff 
development  tool. 


Les  seuils-reperes 


Etat  de  la  question 


Marielle  Simon 

Directrice  Centre  regional  d'  Ottawa 
Institut  d’ etudes  pedagogiques  de 
l’ Ontario 


Le  concept  dc  seuils-reperes  tel  quc 
l’enlend  le  ministbrc  de  l’Education  dc 
l’Ontario  esl  apparu  lors  de  la  rcfontc  cn 
Education,  6 la  suite  dc  l’ctudc  dcs  rdsul- 
tats  des  61cves  ontariens  aux  enquctcs 
provincialcs,  nationalcs  ct  intcrnationalcs 
des  apprcntissagcs.  Les  scuils-repcrcs  sc 
veulcnl  dcs  «poinls  dc  r6f6rcncc»  pcrmct- 
tant  d’assurcr  cohdrcncc,  continuild  cl 
uniformitd  dans  l’enscigncmcnt  cl  dc 
prdciscr  le  dcgrd  ou  la  qualild  du  rcndc- 
mcnt  dcs  616 vcs  dc  la  province  par  rap- 


port aux  attentes  dcs  milieux  scolaire  et 
d’affaires. 

Lc  present  article  fait  etat  de  la  ques- 
tion des  seuils-reperes  provinciaux  en 
mathematiques  pour  la  fin  dc  la  3e,  6e  ct 
9e  anndc  ct  dccrit  certaines  initiatives 
rdgionales  cn  Ontario  frangais.  Pour  ce 
fairc,  nous  avons  d’abord  consulte  Mon- 
sieur Gerald  Hurtubise,  agent  d’dduca- 
tion  au  bureau  regional  d’Ottawa  du 
Ministbrc  d’Education  dc  l’Ontario  et 
rcsponsablc  du  dossier  cn  question.  Nous 
avons  aussi  visionnd  l’dmission  P6da- 
gogic  2000  tdlcdiffusdc  cn  novembre 
1992,  qui  mettait  cn  vedette  Monsieur 
Hurtubise  et  Madame  Constance  Legen- 
lil,  coordonnatricc  des  programmes  du 
Conseil  des  dcolcs  sdpardes  d’Esscx. 
Nous  avons  demandd  6 des  rep  r6  so  ti- 


tan t s/e  s des  conseils  scolaires  et  des  sec- 
tions de  langue  frangaise  de  la  region  de 
Test  de  l’Ontario  de  bien  vouloir 
partager  avec  nous  les  demarches 
entreprises  dans  le  domaine. 

Seuils-reperes  provinciaux 

L’etablissement  de  seuils-rep6res  sert  6 
atteindre  quatre  buts.  Ils  permettent  de 
rassurer  le  public  ontarien  de  la  qualild 
des  programmes-cadres  du  Minist6re  de 
l’Education  de  l’Ontario;  ils  guident 
1’elaboration  des  enquetes  provinciales; 
ils  amcncnt  l’enscignant/e  6 situer  l’616ve 
dans  son  chcmincmcnt  et  foumissent  aux 
enseignants/es  une  vari6l6  d’outils  et  de 
pistes  permettant  d’ajustcr  les  pratiques 
p6dagogiqucs. 

Lc  minist&re  dc  l’Education  s’est 
d’abord  pcnch6  sur  l’61aboration  des 
seuils-rcp6res  provinciaux  cn  maihlma- 
tiques.  Lc  processus  d’61aboraiion  ct  de 
validation  comprcnait  tmis  6 tapes: 

a)  Misc  sur  pied  d’6quipes  dTMaboraiion 

dcs  seuils-r6p6rcs 

Un  groupc  d’enseignants/es  dc  la 
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Copies  still  available  at  $5.35  (GST 
included) 

Or  Four  Special  Theme  Issues  for 
1993  for  only  $32.10  (including  GST) 

Mail  your  order  to  Orbit,  P.O.  Box 
10,  Station  F,  Toronto  M4Y  2L4 
Fax  (416)  696-8409 

ORDER  NOW! 


region  de  Windsor  a elabore  les  seuils- 
reperes  provinciaux  en  mathematiques 
3e  annee.  Un  groupe  d’enseignants/es  de 
la  region  de  North  Bay  et  Timmins  a 
redige  ceux  de  mathematiques  6e  annee. 
Un  groupe  d’enseignants  de  la  region 
d’ Ottawa  s’est  occupe  de  developper  les 
seuils-reperes  mathematiques  9e.  Une 
fois  que  les  indicateurs  furent  identifies 
de  fagon  parallele  dans  les  deux  commu- 
nautes  linguistiques,  il  y a eu  fusion- 
nement  des  deux  etudes. 

b)  Etude  critique  des  seuils-reperes  des 
Conseils  scolaires  de  l’Ontario,  des 
programmes-cadres  et  des  program- 
mes d’etudes  des  autres  provinces 
canadiennes  et  de  plusieurs  pays  occi- 
dentaux  et  orientaux 

Les  equipes  comprenaient  des  en- 
seignants/es,  des  membres  dissociations 
dans  les  domaines  d’ education  et  du  tra- 
vail ainsi  que  des  personnes-ressource 
competentes  en  mathematiques. 

Bien  que  les  seuil-reperes  se  definis- 
saient  a partir  des  pratiques  d’enseigne- 
ment  precedentes  et  actuelles  en 
mathematiques,  on  a tenu  a elargir  les 
cadres  consultatifs  pour  faire  en  sorte 
que  les  seuils-reperes  soient  a la  hauteur 
des  attentes  locales  et  universelles. 

c)  Processus  de  consultation  et  de  vali- 
dation aupres  des  divers  groupes 
interesses 

Une  fois  les  criteres  formules,  le  min- 
ister de  l’Education  de  l’Ontario  dis- 
tribuait  une  ebauche  a tous  les  groupes 
interesses  pour  fins  d’examen  critique. 
Cet  exercice  permettait  d’orienter  les 
politiques  ministerielles  et  de  reviser  les 
seuils-reperes  de  sorte  qu’ils  soient  com- 
plets  et  pertinents. 

Les  seuils-reperes  provinciaux  ont 
trois  volets.  Une  premiere  serie  de  seuils 
sert  a identifier  ce  que  tous  les  eleves 
d’un  niveau  scolaire  devraient  etre  capa- 
ble de  faire  a la  fin  de  l’annee  scolaire. 
Une  deuxieme  vise  la  plupart  des  eleves 
tandis  que  pour  la  troisieme  serie,  on 
s’attend  a ce  que  quelques  eleves  seule- 
ment  manifestent  les  comportements 
decrits.  Cette  marge  de  manoeuvre 
respecte  les  habiletes  de  tous  les  eleves. 

On  prevoit  plusieurs  retombees  de 
l’utilisation  des  seuils-reperes.  Au  niveau 
provincial,  ils  meneront  a un  enseigne- 
ment  des  mathematiques  plus  coherent  et 
uniforme.  Ils  renseigneront  la  clientele  de 


ce  qui  est  attendu  des  eleves  au  niveau 
des  connaissances,  des  competences  et 
des  attitudes  a certains  moments  fixes. 
Finalement,  les  conseils  scolaires  en 
province  pourront  s’inspirer  des  seuils- 
reperes  provinciaux  lors  de  l’elaboration 
de  strategies  locales  d’enseignement  et 
devaluation.  On  espere  que  Tutilisation 
des  seuils  amenera  un  changement  dans 
la  fagon  de  penser  vis-a-vis  l’enseigne- 
ment  des  mathematiques.  II  ne  suffira 
plus  de  suivre  fidelement  et  uniquement 
un  manuel  scolaire  ou  un  programme 
d’etude  mais  de  planifier  l’enseignement 
de  sorte  que  tous  les  eleves  possedent  les 
savoirs,  les  savoirs-faire  et  les  savoirs- 
etre  desires. 

On  se  penche  actuellement  sur  l’elab- 
oration de  seuils-reperes  provinciaux  en 
communication  ecrite  et  orale. 

Seuil-repgres  locaux 

Plusieurs  conseils  scolaires  ou  sections 
de  langue  frangaise  en  Ontario  ont  deja 
elabore  des  seuils-reperes  locaux  dans 
certaines  matibres.  Dans  l’est  de 
l’Ontario  certains  conseils  ou  sections 
ont  etabli  entre  autres,  des  seuils  en 
chimie  cycle  superieur,  en  biologie  cycle 
superieur,  en  mathematiques  9e  annee, 
en  anglais/English  10e  annee  niveau 
general  et  en  frangais  cycle  moyen. 

Dans  un  conseil  en  particulier,  par 
exemple,  une  equipe  de  specialistes  a 
ceme  d’abord  les  dimensions  essentielles 
des  programmes  d’etudes.  On  a ensuite 
identifie  les  criteres  propres  a chaque 
dimension  et  definit  la  performance  min- 
imale attendue  a partir  de  l’analyse  cri- 
tique des  travaux  des  eleves  et  des 
epreuves  d’ evaluation.  On  a aussi  foumi 
des  exemples  de  questions  ou  d’instru- 
ments  pour  chacun  des  seuils.  Ces  seuils 
ont  ete  mis  a l’essai  pendant  un  an. 

Dans  un  autre  conseil  la  participation 
a des  ateliers  en  evaluation  des  appren- 
tissages  a amene  les  conseillers/eres  ped- 
agogiques  a concevoir  les  seuils-reperes 
en  termes  d’indices  ou  d’indicateurs  de 
reussite. 

Dans  la  plupart  des  conseils  scolaires 
cependant,  on  prdfere  attendre  la  mise  en 
oeuvre  officielle  des  seuils-reperes 
provinciaux  et  du  programme  d’etudes 
commun  de  la  le  a la  9e  annee.  On  ela- 
borera  ou  adaptera  alors  les  programmes 
d’etudes  ainsi  que  les  strategies 
d’enseignement  et  devaluation  a la 
lumiere  de  ces  documents. 
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How  Will  the  Public 
Respond  to  Provincial 
Benchmarks? 

Some  Lessons  from  Public  Opinion  on  Testing 


Doug  Hart 

Senior  Research  Officer, 

OISE  Survey  of  Educational  Issues 


An  important  goal  of  the  Provincial 
Standards  (Benchmarks)  program  (and 
local  Board  programs)  is  to  raise  public 
confidence  in  the  school  system. 
Whether  this  succeeds  will,  in  large  part, 
depend  on  the  extent  to  which  bench- 
marks are  seen  by  the  public  as  increas- 
ing the  accountability  of  the  school 
system.  At  present,  there  is  virtually  no 
information  on  how  the  public  at  large 
(as  opposed  to  organized  interest  groups) 
will  react  to  the  provincial  approach.  In 
the  absence  of  direct  data,  however,  it  is 
worth  re-examining  public  opinion  on 
issues  of  testing  for  clues  to  likely  public 
reaction  to  benchmarks.  Most  of  the  fol- 
lowing is  based  on  opinion  data  from  the 
OISE  Surveys  of  Educational  Issues, 
conducted  every  two  years  and  last 
administered  in  September,  1992.1 

In  predicting  public  response  to  the 
benchmarks  initiative,  four  lessons  can 
be  drawn  from  the  record  of  public  opin- 
ion on  testing  issues: 

1.  Benchmarks  will  be  readily  accepted 
insofar  as  they  are  associated  in  the 
public  mind  with  province-wide  test- 
ing. 

2.  Benchmarks  will  encounter  wide- 
spread opposition  if  they  arc  seen  as 
supplanting  or  seriously  compromis- 
ing local  autonomy  in  student  assess- 


ment and,  in  particular,  the  role  of 
teachers. 

3.  Only  a minority  of  the  public  will  hold 
opinions  which  are  “consistent”  — by 
the  standards  of  organized  interest 
groups  — across  issues  of  bench- 
marks, role  of  teachers,  control  over 
curriculum,  province-wide  testing,  and 
so  on.  Thus,  while  at  the  level  of  pub- 
lic debate  support  for  local  control  of 
curriculum  typically  goes  together 
with  opposition  to  province-wide  tests, 
opinions  on  these  issues  may  be  only 
weakly  correlated  among  the  mass 
public. 

4.  Parents  and  non-parents  will  make 
similar  judgments  about  benchmarks, 
suggesting  that  long-standing  orienta- 
tions to  assessment,  rather  than  direct 
interest  and  experience  with  the 
school  system,  are  pivotal  in  opinion 
formation  on  assessment  issues. 

That  benchmarks  can  be  readily  sold 
to  the  public  if  packaged  as,  or  with, 
province-wide  testing  will  undoubtedly 
be  seen  as  obvious  by  most  readers.  It  is 
hardly  news  that  polls  have  consistently 
found  strong  majority  support  for  testing. 
Two  recent  surveys  confirm  the  pattern. 
An  Angus  Reid  poll  released  in  early 
February  found  that  77  percent  of 
Ontario  respondents  (and  76  percent 
nationally)  would  support  “...  standard- 
ized testing  in  our  educational  system” 
(Toronto  Star,  1993,  p.  A 16).  And  in 
September,  1992,  73  percent  of  respon- 
dents to  the  OISE  Survey  of  Educational 


Issues  agreed  that  “province-wide  tests 
should  be  used  to  assess  the  individual 
performance  of  elementary  school  stu- 
dents.” This  represented  a sharp  increase 
from  1990,  when  the  figure  was  59  per- 
cent. The  1992  OISE  Survey  also  found 
that  support  for  testing  was  particularly 
high  among  those  who  thought  that  the 
most  useful  information  to  parents  was 
how  their  child’s  skills  stacked  up 
against  official  standards  or  benchmarks. 
Note  that  “benchmarks”  were  defined 
generally  in  the  survey  as  official  stan- 
dards of  what  skills  should  have  been 
learned.  The  questionnaire  items  did  not 
refer  to  the  provincial  benchmark  initia- 
tive explicitly.  See  Figure  2 for  the  exact 
wording  of  the  items. 

There  is,  however,  more  to  public 
views  on  province-wide  testing  than 
these  simple,  and  much  publicized, 
results.  While  an  overwhelming  majority 
support  province-wide  testing,  only  a 
very  small  minority  want  such  tests  to  be 
the  most  important  means  of  assessing 
students.  In  1990,  only  9 percent  of 
respondents  to  the  OISE  Survey  wanted 
provincial  tests  to  be  most  important  in 
judging  the  academic  progress  of  ele- 
mentary school  students.  (This  question 
was  not  asked  in  1992.)  Similarly  small 
minorities  in  the  1979,  1982,  and  1988 
OISE  Surveys  had  supported  giving  pri- 
ority to  provincial  tests  in  assessing  stu- 
dents “in  the  higher  grades"  (Livingstone 
et  al.,  1989,  p.  21).  In  1988,  these  views 
on  the  best  means  of  assessing  students 
co-existed  with  majority  support  for  the 
use  of  province-wide  tests.  A survey 
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conducted  early  in  1991  by  the  Institute 
for  Social  Research  (ISR),  York  Univer- 
sity, found  that  only  13  percent  thought 
students’  progress  should  be  judged  by 
province-wide  tests  alone  (ISR,  1991,  p. 
4).  This  option  is  admittedly  more 
extreme  than  the  position  taken  by  most 
proponents  of  testing. 

Thus,  the  second  lesson  to  be  learned 
from  opinion  research  on  testing  is  that 
support  for  province-wide  testing  does 
not  translate  into  an  identification  of 
accountability  with  meeting  externally 
set  standards.  Issues  of  local  input  and 
influence  on  student  assessment  are  also 
important.  Both  the  OISE  survey  and 
ISR  questions  juxtaposed  province-wide 


tests  with  local  options;  in  the  ISR  ques- 
tion, the  alternative  was  “parents  and 
teachers”  as  "...  best  to  judge  a student’s 
progress.”  Twenty-nine  percent  selected 
this  option;  52  percent  opted  for  both 
province-wide  tests  and  parents  and 
teachers.  The  options  for  the  OISE  ques- 
tion are  shown  in  Figure  1. 

Just  over  half  of  respondents  support 
a wholly  locally  based  option  as  the  best 
way  to  assess  elementary  students’  aca- 
demic progress.  The  proportions  were 
similar  in  1979,  1982,  and  1988  when 
the  question  was  asked  about  students  in 
the  higher  grades.  One-third  think  that 
the  best  way  to  assess  progress  involves 
joint  local/provincial  responsibility, 


while  only  a small  minority  would  hand 
the  job  to  the  province  alone.  While  the 
majority  of  the  public  are  wedded  to  the 
test  format  as  the  best  means  of  student 
assessment,  local  teacher-made  tests 
alone  or  in  conjunction  with  province- 
wide tests  are  seen  as  key. 

A majority  of  the  public  are  strongly 
supportive  of  the  role  of  local  teachers  in 
student  assessment  yet  they  also  favour 
the  use  of  province-wide  tests.  These 
positions  do  not  usually  go  together  in 
the  polarized  debates  over  testing.  Thus, 
the  third  lesson  to  be  learned  from  public 
opinion  on  testing  is  that  the  public  often 
holds  positions  on  issues  which  do  not  go 
together  “logically,”  at  least  according  to 
those  at  the  centre  of  public  debates. 

In  1984,  the  OISE  Survey  first  asked 
whether  province-wide  tests  should  be 
used  to  assess  the  individual  performance 
of  high  school  students.  The  question  was 
immediately  preceded  by  one  on  who 
should  have  the  most  influence  on  what 
is  taught  in  local  schools.  Respondents 
were  almost  equally  split  over  whether 
local  groups  (school  board,  local  teach- 
ers, parents)  or  the  provincial  or  federal 
government  should  have  the  most  influ- 
ence over  the  curriculum.  Those  who 
chose  one  of  the  local  options  were  less 
likely  to  support  use  of  province  wide- 
tests  than  others,  but  the  difference  was 
narrow:  63  versus  73  percent.  In  other 
words,  a substantial  majority  of  those 
who  favoured  a dominant  local  influence 
on  curriculum  also  favoured  province- 
wide tests,  which  are  usually  regarded  in 
public  debates  as  an  instrument  of  cen- 
tralized control  over  what  is  taught  in 
local  schools. 

The  1992  OISE  Survey  included  a 
more  general  question  on  whether  the 
involvement  of  various  groups  in  the 
operation  of  local  schools  should  in- 
crease, decrease,  or  stay  the  same.  Many 
more  respondents  favoured  the  increased 
involvement  of  parents  (84  percent)  and 
teachers  (69  percent)  than  the  increased 
involvement  of  the  Ministry  of  Educa- 
tion (41  percent)  and  school  board 
administrators  (39  percent).  In  other 
words,  support  was  highest  for  those  in 
immediate  contact  with  students.  Views 
on  what  should  happen  to  the  involve- 
ment of  different  groups  in  the  operation 
of  local  schools  had  no  bearing  on  opin- 
ion regarding  province-wide  testing. 
Those  who  wanted  more  involvement  of 
teachers  and  parents  in  the  operation  of 
local  schools  were  no  less  likely  than 


Figure  1 “Which  one  of  the  following  do  you  think  is  the  best  way  of  judging  the 
academic  progress  of  students  in  elementary  school?”  ( Asked  in  1990) 


All  respondents 
% 

Parents 

% 

Non-parents 

% 

The  individual  teacher  should 
have  sole  responsibility  for 
making  assessments 

20 

20 

20 

Assessments  should  be  based 
on  tests  set  jointly  by  teachers 
in  the  school 

34 

37 

31 

Provincial  tests  should  be  used 
to  supplement  teacher-made  tests 

35 

34 

35 

Provincial  tests  should  be  most 
important 

9 

7 

11 

Don't  know/  not  stated 

3 

3 

3 

Figure  2 Support  for  Use  of  Province-wide  Tests  at  the  Elementary  Level  by  Views 
on  Most  Useful  Information  to  Parents  of  Elementary  School  Students 


Most  useful  information  to  parents 
% agreeing  with  province-wide  tests 

All 

respondents 

Parents 

Non-parents 

Number  of 
respondents* 

How  their  child’s  skills 
compare  to  how  they 
were  doing  earlier 

64 

63 

65 

491 

How  their  child’s  skills 
compare  to  those  of 
other  students 

76 

73 

77 

160 

How  their  child’s  skills 
compare  to  official 
standards  a benchmarks 

85 

85 

88 

288 

Don't  know / not  stated 

75 

54 

82 

62 

* The  sample  included  396  parents  of  elementary  and  lor  high  school  students. 
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others  to  support  province-wide  testing; 
those  who  favoured  more  Ministry 
involvement  were  no  more  likely  to  indi- 
cate support  for  testing. 

The  1992  Survey  also  asked  respon- 
dents to  select  the  single  most  useful 
information  to  parents  regarding  how 
their  child  is  doing  in  school.  The  options 
are  shown  in  Figure  2.  Those  who  select- 
ed the  option  which  focussed  on  a child’s 
progress  in  relation  to  his  or  her  own  past 
performance  were  least  likely  to  support 
province-wide  testing.  However,  whether 
respondents  held  child-centred,  norm- 
oriented,  or  benchmark-oriented  priori- 
ties, a majority  supported  province-wide 
tests.  This  is  the  case  for  both  parents  and 
non-parents.  Those  who  selected  child- 
centred  assessment  comprised  over  40 
percent  of  those  supporting  province- 
wide testing. 

That  public  views  are  frequently,  even 
typically,  “inconsistent”  when  judged  by 
the  standards  of  opinion-leaders  has  been 
a common  finding  of  opinion  research. 
From  one  perspective,  this  reflects  the 
“inattentiveness”  of  mass  publics  to 
views  advanced  by  opinion-leaders.  From 
another,  inconsistencies  are  the  counter- 
part of  overly  rigid  positions  which  have 
evolved  in  an  increasingly  polarized 


debate.  In  either  case,  these  patterns 
counsel  caution  in  interpreting  the  reac- 
tion of  public  opinion  to  provincial  stan- 
dards and  benchmarks.  It  is  important  to 
note  that  new  testing  initiatives  will  like- 
ly be  evaluated  against  what  appear  to  be 
quite  stable  and  complex  general  orienta- 
tions to  student  assessment. 

This  leads  to  our  final  point.  It  is 
unlikely  that  the  views  of  organized  inter- 
est groups  on  benchmarks  will  simply  be 
adopted  by  large  segments  of  the  public. 
Yet  general,  abstracted  orientations  to 
assessment  rather  than  concrete  interests 
and  recent  direct  experience  with  the 
school  are  likely  to  be  pivotal  in  opinion 
formation.  This  is  suggested  by  the  fact 
that  despite  their  much  more  direct  con- 
tact and  intensive  involvement  with  the 
school  system,  the  views  of  parents  of 
elementary  and  high  school  students  dif- 
fer little  from  those  of  non-parents  on 
assessment  issues.  Parents  are  as  likely  to 
favour  use  of  province-wide  tests  as  non- 
parents. As  noted  in  Tables  1 and  2,  the 
distribution  of  views  on  the  best  method 
of  assessing  students’  academic  progress 
and  the  most  useful  information  to  par- 
ents about  their  child’s  progress  are  simi- 
lar for  parents  and  non-parents.  “Don’t 
know”  and  non-response  rates  on  assess- 


ment questions  are  low  for  both  groups. 
That  neither  direct  experience  nor  “atten- 
tiveness” to  organized  interest  groups  will 
play  a key  role  in  attitude  formation  sug- 
gests that  much  of  the  public  will  attempt 
to  put  whatever  “new  wine”  comes  with 
benchmarks  into  “old  bottles.” 


NOTE 

1.  The  nine  OISE  surveys  have  all  been  published 
and  distributed  by  OISE  Press.  They  are  as  follows: 
D.  W.  Livingstone.  Public  Altitudes  Toward  Edu- 
cation in  Ontario  197 8\  D.  W.  Livingstone  and  D. 
Hart.  Public  Attitudes  Toward  Education  in 
Ontario  1979,  1980.  D.  W.  Livingstone,  D.  Hart, 
and  L.  D.  McLean.  Public  Altitudes  Toward  Edu- 
cation in  Ontario  1982.  D.  W.  Livingstone,  D. 
Hart,  and  L.  E.  Davie.  Public  Attitudes  Toward 
Education  in  Ontario  1984,  1986,  1988,  1990,  and 
1992.  For  more  detailed  information  about  the 
design  of  the  OISE  survey  series,  see  especially  D. 
W.  Livingstone,  D.  Hart,  and  L.  E.  Davie  (1985). 
Public  Altitudes  Toward  Education  in  Ontario 
1984:  Fifth  OISE  Survey.  Toronto:  OISE  Press. 
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A Business  Perspective  on 
Standards  and  Accountability 

An  Interview  with  Jim  Poole,  Ontario 
Chamber  of  Commerce 


Jim  Poole 

Member  of  the  Education  Policy 
Committee  of  the  Ontario  Chamber 
of  Commerce  and  Representative  on 
the  Assessment  Advisory  Committee 
of  the  Learning  Assessment  Branch, 
Ontario  Ministry  of  Education 
and  Training 

Interviewed  by  Howard  Russell 


Russell  What  do  you  think  the  Ontario 
Chamber  of  Commerce  wants  from  edu- 
cation authorities  in  order  to  satisfy  the 
need  for  accountability? 

Poole  The  Chamber  of  Commerce 
wants  an  education  system  for  Ontario 
which  is  accountable,  and  the  best  way 
to  show  this  is  through  benchmarks,  that 
is,  performance  standards  and  achieve- 
ment levels,  which  are  reported  publicly. 
The  largest  problem  is  the  lack  of  infor- 
mation, both  from  a reporting  sense,  as 
well  as  a perceived  lack  of  information 
generated  within  the  system  to  provide 
feedback  to  our  teachers,  educators,  and 
administrators  to  help  improve  the  sys- 
tem. The  business  community  wants  to 
receive  graduates  to  help  compete  glob- 
ally against  nations  that  presently  score 
higher  than  Canada  on  international  tests 
and  it  supports,  as  a first  step  towards 
accountability,  regular  formal  assess- 
ment of  student  progress  and  publication 
of  overall  results  in  core  subjects  through 
standardized  provincial  systems.  These 
should  be  put  in  place  at  the  earliest 
stages  of  elementary  education. 

To  get  going  on  that,  we  need  a better 
definition  of  curriculum  and  core  sub- 
jects starting  in  the  primary  years  and 
then  we  need  to  measure  progress  to- 
wards mastery  of  the  curriculum. 


Russell  Do  you  see  evidence  of  pro- 
gress towards  the  satisfaction  of  this 
need? 

Poole  Where  I see  the  evidence  is  in 
the  direction  taken  by  the  Ministry  of 
Education  and  Training  through  the 
provincial  benchmarks  — now  called 
“standards”  — program,  which  has  been 
running  on  a number  of  subjects  for  at 
least  the  last  five  years.  It  provides  infor- 
mation for  educators  in  the  system  after 
Grades  3,  6,  and  9.  That  is  excellent 
feedback  to  give  to  education  system 
managers  so  they  can  help  improve  the 


— I believe 
too  many 
groups  have 
their  fingers 
in  the 

education  pie. 

- Jim  Poole 


system.  I believe  it’s  important  to  decide 
on  a task,  look  at  the  results,  and  imple- 
ment the  learning  into  the  system,  and 
then  try  to  continuously  improve  the  pro- 
cess. Our  committee  has  been  supporting 
the  Ministry  on  this.  I believe  that  the 
difficulty  is  for  parents  and  business  — 
they  feel  it’s  not  far  enough,  not  fast 
enough,  not  detailed  enough,  especially 
in  the  primary  years.  Yet  we’re  encour- 
aged by  the  number  of  professionals  in 
the  system  who  want  information, 


accountability,  and  freedom  to  help 
improve  the  system. 

Russell  Is  the  benchmarks  or  stan- 
dards thrust  distinguished  from  others  in 
the  past  with  respect  to  the  clarity  of 
goals  and  the  outcomes  of  education? 
I’m  thinking  now  of  standardized  tests 
with  the  label  “math”  or  “literacy”  asso- 
ciated with  school  marks. 

Poole  The  business  community,  par- 
ents, and  all  other  stakeholders  in  the 
system  will  know  more  if  the  results  are 
communicated  properly  through  all 
channels.  I believe  one  of  the  difficulties 
is  that  the  system  goes  off  and  takes  the 
information  into  itself  and  doesn’t  com- 
municate it  outside,  and  we’re  left  with  a 
vacuum  of  knowledge  that  is  currently 
raising  people’s  concerns.  It’s  obvious 
there  are  barriers  to  progress,  because 
results  from  system  change  haven’t  been 
achieved  to  date,  and  I believe  that’s 
because  too  many  groups  have  their  fin- 
gers in  the  education  pie.  This  leads  to  a 
lack  of  trust  among  participants  in  the 
process.  What  is  needed  is  to  set  bench- 
marks or  standards  and  get  information 
flowing  so  the  system  can  use  the  infor- 
mation to  improve.  In  my  view,  there  are 
a lot  of  opinions  floating  around,  but  not 
enough  facts.  With  information,  the  level 
of  trust  will  certainly  build,  reduce  barri- 
ers, and  help  speed  change.  The  Cham- 
ber of  Commerce  wants  to  help  in  this 
regard  by  building  consensus  between 
labour,  education,  and  business  in  a non- 
partisan setting.  We  believe  the  politi- 
cization of  the  education  process,  which 
changes  with  each  government  and  Min- 
ister, is  in  fact  a barrier  to  progress. 
Politicians  are  involved  and  are  using  it 
to  further  the  political  process.  We  think 
education  should  rise  above  that  and  be  a 
longer  term  direction  of  the  province. 

Russell  Does  the  business  communi- 
ty use  benchmarks  or  indicators?  Your 
experience  in  the  business  community 
may  tell  us  something  about  how  trust 
can  be  built. 

Poole  The  education  community,  I 
think,  goes  through  a similar  process. 
You  plan  the  curriculum.  You  have  a 
longer  term  need.  You  introduce  the  cur- 
riculum. You  check  to  see  how  students 
are  achieving  and  then  you  make  changes 
in  whatever  the  delivery  process  is  to 
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improve  results.  I think  that  process  is 
very  similar  to  the  process  in  business  to 
measure  progress  and  track  performance 
towards  a goal.  The  business  community 
learned  that  it  exists  only  to  serve  the 
customer,  and  successful  companies  do 
everything  to  help  their  front-line 
employees  delight  their  clients.  Business 
tries  to  get  the  right  information  to  these 
workers  so  they  understand  their  achieve- 
ment and  are  accountable  for  continuous 


The  business  community 
wants  to  receive 
graduates  to  help 
compete  globally 
against  nations  that 
presently  score  higher 
than  Canada  on 
international  tests 


progress  towards  goals  that  they  help  to 
set.  There’s  ownership  of  these  goals. 
Many  businesses  seek  feedback  or  survey 
their  customers  or  consumers  of  services 
so  they  can  “benchmark”  progress 
towards  improvement. 

How  does  this  apply  to  the  education 
system?  I believe  educators  should 
understand  the  needs  of  students  and 
graduates  and  the  customers  of  the  edu- 
cation system  — business,  universities, 
community  colleges,  even  drop-outs  of 
the  system;  these  are  just  dissatisfied 
consumers.  This  will  help  educators  gain 
knowledge  about  the  performance  of  the 
education  system. 

Russell  In  education,  we  begin  with 
some  lack  of  trust  between  educators  and 
their  clients  and  we  may  end  up  eroding 
the  trust  that  exists  when  the  benchmark 
information  — the  very  kind  of  informa- 
tion educators  need  to  improve  the  ser- 
vice they  provide  — is  distributed  in  a 
way  that  is  misleading  but  newsworthy. 
Does  this  kind  of  erosion  take  place  in 
the  business  community? 

Poole  The  issue  of  not  communicat- 
ing fully  is,  in  my  mind,  the  base  issue 
behind  the  erosion  of  trust  throughout 
the  system.  Without  real  data  and  infor- 
mation, the  partners  in  the  education  sys- 
tem become  suspicious  of  each  other’s 


motives.  You  get  many  that  are  trying 
very  hard  to  do  good  work  but  that 
counts  for  nothing  if  it  is  not  communi- 
cated properly.  I think  that  points  to  an 
alliance  between  all  of  the  stakeholders 
in  the  system  to  share  common  learnings, 
if  I can  put  it  in  general  terms  here. 

Russell  What  are  the  most  important 
features  of  a benchmarks  program  or 
what  are  the  features  that  must  be  pre- 
served at  all  cost? 

Poole  The  most  important  features  of 
the  benchmark  program  are,  first,  that  it 
communicates  progress  to  the  public, 
including  business,  and  that’s  how  we 
raise  confidence  in  the  system.  The  sec- 
ond is  that  it  provides  feedback  to  the 
system  so  professionals  can  improve  the 
education  progress.  The  third  is  that  it 
gives  feedback  to  individual  students  and 
parents  against  a set  of  meaningful  stan- 
dards, by  grade,  so  that  parents  under- 
stand what  their  children  are  capable  of. 
I think  those  are  three  things  that  have  to 
be  preserved.  If  something  had  to  be 
eliminated  due  to  cost,  I would  suggest 
individual  testing  for  every  student  could 
be  eliminated  and  we  could  use  statisti- 
cal sampling  techniques  to  cut  costs  with 
the  same  accuracy  of  data. 

Russell  Do  you  see  a connection  be- 
tween educational  data  and  workplace 
data  when  we  move  in  education  towards 
benchmark-level  data  as  opposed  to 
macro-level  data? 

Poole  When  we  collect  data,  the  best 
possible  indicator  of  achievement  is 
something  that  is  knife-edged  — either 
you  did  it  or  you  didn’t.  We  work  very 
hard  to  get  indicators  that  give  that 
progress.  Getting  data  to  track  progress 
towards  a plan  and  to  evolve  an  organi- 
zation or  process  towards  a desired  out- 
come is,  in  fact,  the  kind  of  measure  we 
in  business  apply  to  our  processes,  and  I 
see  the  use  of  similar  statistical  data  as 
being  supportive  in  both  education  and 
business  communities.  The  effort  to  gain 
an  understanding  of  the  system  before 
the  action  was  taken,  and  then  after,  and 
determining  how  well  the  curriculum 
passed  through  to  the  student  is  exactly 
the  kind  of  statistical  evidence  you  re- 
quire in  order  to  manage  the  education 
system;  you  need  the  feedback  for  teach- 
ers to  improve  the  process. 


Russell  Do  you  have  any  advice  to 
teachers  who  are  tempted  to  begin  work 
with  benchmarks? 

Poole  My  advice  is  to  start.  No  bench- 
mark system  is  going  to  be  perfect  the 
first  time.  So,  getting  started  on  it  — 
understanding  the  system,  improving  the 
benchmarking  system  with  feedback  to 
the  originators  — is  the  best  place  to 
start.  You  end  up  with  a much  improved 
system  with  that  kind  of  experience.  To 
work  towards  achievement  of  real  data 
through  benchmarking,  or  setting  stan- 
dards, and  keeping  records  of  overall 
achievement  and  then  to  work  to  im- 
prove the  accuracy  and  achievement  of 
this  process  is  what  I think  educators 
should  be  driving  towards.  When  busi- 
ness uses  benchmarks  — and  we  do 
benchmark  processes  — it’s  not  to  com- 
pare numbers  but  to  first  benchmark  the 
process  in  order  to  understand  it  and  to 
lead  to  a better  result. 

Russell  Let’s  talk  a bit  about  that.  Do 
we  have  enough  clarity  about  what  we 
mean  by  benchmark  and  is  it  going  to  be 
intelligible  to  the  business  community? 

Poole  When  we  talk  about  bench- 
marking, we  talk  about  going  into  a com- 
pany that  has  a similar  function  and 
looking  at  a process  — billing  to  the  cus- 
tomer, for  example  — at  how  it  works, 
what  are  the  benefits,  what’s  good  and 
bad,  and  then  we  turn  around  and  look  at 
our  process,  put  it  up  side  by  side,  and 
see  whether  there  are  areas  of  improve- 
ment in  process;  then  we  measure  out- 
comes towards  that.  I don’t  believe  the 
theory  is  any  different  in  the  education 
community.  I believe  that  what  we  call  a 
“benchmark”  is  the  measurement  of  out- 
come of  the  process. 

Russell  Any  concluding  comments? 

Poole  The  Ontario  Chamber  of  Com- 
merce believes  very  strongly  that  the 
benchmark  or  standards  program  in  the 
Ministry  of  Education  and  Training  is  a 
step  in  the  right  direction,  that  informa- 
tion flow,  both  internal  and  external,  is 
essential  to  manage  the  education  system, 
and  that  can  be  achieved  through  infor- 
mation generated  by  benchmarks.  The 
Ontario  Chamber  of  Commerce  wants  to 
help  build  consensus  towards  action 
between  education,  labour,  and  business. 
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Moving  in  a New 
Direction 


An  Interview  with  the  Director  and  the  Superintendent 
of  Curriculum,  Toronto  Board  of  Education 


Russell  I’d  like  to  begin  by  asking 
you  to  describe  the  Toronto  Board  bench- 
mark program. 

Green  We  define  our  benchmarks  as 
strategies  or  tools  to  help  us  establish 
standards  of  achievement  for  students  in 
our  board.  We  see  benchmarks  as  pro- 
viding examples  of  the  kinds  of  perfor- 
mances that  lead  to  academic  success  as 
well  as  system-wide  reference  points  to 
assist  our  teachers  in  student  evaluation. 
All  of  us  involved  in  education  these 
days  believe  that  we  need  to  make  both 
what  we’re  doing  and  the  results  more 
readily  understandable  by  all  concerned, 
particularly  by  the  parents  and  the  care- 
givers of  the  children  in  our  schools.  It’s 
no  longer  enough  for  us  to  understand 
what  children  are  achieving.  It  is  abso- 
lutely paramount  that  parents  and  the 
children  themselves  have  a clear  notion 
of  how  they’re  doing  relative  to  others. 
Our  benchmarks  are  helping  us  move  in 
that  direction. 

Russell  I see  the  Toronto  Board 
Benchmarks  as  marking  a fundamental 
shift  in  the  nature  of  accountability  data. 
You’ve  taken  a lot  of  heat  for  moving  in 
that  new  direction. 

Green  Yes,  we’ve  paid  a big  price 
both  in  terms  of  actual  dollars  and  in 
human  resources.  We  believe  it’s  a price 
well  invested.  We  think  we’ve  developed 
an  eclectic  array  of  strategies  that  capture 
many  of  children’s  capacities  and  reflect 
their  ability  to  perform  in  a wide  range 
of  areas,  not  just  in  a narrow  band.  I 
think  we  all  know  that  as  we  look  at  the 
demands  of  a global  economy  and  the 
demands  of  living  in  the  21st  century, 
there  is  going  to  be  no  single  track  for 


performance.  It  will  be  a requirement  for 
young  people  to  take  their  knowledge 
from  one  context  to  another  with  ease 
and  to  operate  in  a problem-solving 
mode.  We  believe  very  strongly  that 
what  we’re  doing  and  how  we’re  assess- 
ing that  must  reflect  those  realities  for 
our  kids  as  they  leave  our  schools.  So 
there’s  no  point  in  having  a set  of  evalu- 
ative measures  that  test  only  pen  and 
paper  skills,  or  “fill-in-the-blank”  skills, 
when  we  know  the  world  is  far  more 
complex  than  that. 

Sullivan  I think  it’s  important  for 
people  to  know  that  we  conducted  genu- 
nine  research  to  develop  our  bench- 
marks. We  found  out  what  the  range  of 
performances  was  on  each  individual 
task  and  then  determined  the  percentages 
of  children  that  were  operating  at  each 
level.  The  descriptions  that  are  then 
gained  for  each  one  of  those  levels  of 
performance  become  criteria  for  teachers 
to  judge  future  performances  of  students 
in  their  classrooms. 

Russell  What  progress  have  you 
made  over  this  last  ten-year  period?  Do 
you  see  we’re  near  the  end  of  it,  or  just 
beginning,  or  how  would  you  describe 
it? 

Green  We  see  benchmarks  as  not 
simply  tests.  We  see  them  as  ways  to 
gain  insight  into  student  performance 
and  achievement.  And  we  also  see  them 
as  providing  a framework  within  which 
teachers  can  reflect  on  their  own  pro- 
grams and  develop  a window  on  their 
activity  and  their  methodology.  They’re 
really  intended,  in  the  end,  to  serve  as 
appropriate  models  for  assessment  in 
every  classroom.  In  the  past  five  years. 
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we’ve  tried  to  make  benchmarks  under- 
standable to  all  of  our  leaders  and  to  in- 
dividuals who  are  responsible  for  the 
professional  growth  of  staff  in  every 
school  setting.  We’ve  actually  completed 
one  cycle  of  very  rigorous  in-service  for 
all  of  our  consultative  staff,  for  princi- 
pals and  vice-principals,  and  we’ve  made 
the  implementation  of  benchmarks  site- 
based.  Yet  it’s  a long-term  process  and  I 
suppose  our  current  frustration  is  that 
we’re  not  sure  if  we’ve  managed  to  help 
teachers  translate  what  they’ve  learned 
into  language  that  is  accessible  to  par- 
ents. Our  biggest  challenge  now  is  to 
make  sure  we  get  past  the  jargon  associ- 
ated with  any  professional  language  and 
make  our  benchmarks  clearly  under- 
standable to  parents. 

Russell  It  sounds  like  you’ve  had  to 
grapple  with  a communication  barrier,  or 
at  times  a conceptual  barrier.  I’d  like  you 
to  give  us  a little  more  insight  into  deal- 
ing with  this  problem. 

Green  We’ve  had  to  grapple  with 
getting  a common  understanding  among 
our  teachers.  It  wasn’t  there  to  begin 
with  and  we  had  to  introduce  them  to  a 
very  complex  interconnected  set  of  eval- 
uation strategies.  So  first  we  had  an  edu- 
cation problem  or  challenge  with  our 
own  teachers.  We  had  to  create  all  kinds 
of  opportunities  for  teachers  to  talk,  to 
come  to  that  understanding. 

Sullivan  In  other  words,  in  the  first 
phase  we  tried  to  ensure  that  all  our 
administrators,  curriculum  staff,  and 
teachers  really  understood  the  philoso- 
phy, principles,  and  practices  related  to 
benchmarks.  The  second  phase  really  is 
more  related  to  evaluation  and  to  ensur- 
ing that  we  move  from  benchmarks  as 
good  models  of  classroom  practice  to 
using  benchmarks  in  the  evaluation  of 
daily  work.  Then  there’s  the  reporting  to 
parents  phase.  We’re  conducting  a pilot 
in  mathematics  and  language  in  two  dif- 
ferent families  of  schools  using  a portfo- 
lio system  that  allows  teachers  to  collect 
samples  of  student  work  and  to  track  stu- 
dent progress  based  on  benchmarks.  The 
feedback  from  those  two  pilots  will 
allow  us  to  move  into  a system-wide  im- 
plementation at  some  point  early  next 
year.  Our  early  feedback  from  teachers, 
parents,  and  principals  is  that  they  finally 
sec  how  the  total  program  can  be  used  to 


help  them  understand  what  children  are 
learning  in  the  classroom  and  how  well 
their  students  are  progressing  in  relation 
to  system  standards. 

Russell  How  about  the  community  at 
large?  You’ve  said  that  teachers  under- 
stand the  direction  you  are  taking  but 
what  about  others  in  die  community? 

Green  That’s  the  point  I was  trying  to 
make  a few  moments  ago,  that  we  have  a 
translation  task  before  us  and  particularly 
with  parents  of  children  who  are  disad- 
vantaged in  one  way  or  another.  They 
believe  that  their  children  are  not  having 
the  same  advantages,  outcomes,  or 
results  from  the  school  experience  as 
other  children.  So  they’re  a bit  impatient 
with  the  detail  and  the  obscure  language 
that  sometimes  accompanies  professional 
talk.  We  are  trying  very  hard  to  make 
things  clear.  In  the  last  little  while  we 
developed  a resource  sheet  for  teachers, 
“Benchmarks  and  the  Basics,”  which 
talks  about  what  it  is  that  parents  define 
as  basics.  Some  people  define  the  basics 
with  a sort  of  myopic  nostalgia.  We  are 
trying  to  help  them  see  that  the  basics  are 
given  in  the  Toronto  school  system,  but 
that  what’s  required  in  the  future  will  be 
more  than  basic  skills  and,  therefore,  the 
evaluation  tools  we  use  have  to  assess 
more  than  what  would  be  defined  as  the 
old  basic  skills. 

Russell  Do  minority  groups  in  Toron- 
to have  a distinct  position  on  standards 
and  benchmarks? 

Green  The  leaders  of  our  different 
parent  groups  believe  that  their  children 
should  be  performing  at  the  same  level, 
with  the  same  range  of  opportunities,  as 
other  children  have  traditionally.  So  they 
don’t  want  remedial  programs.  They 
want  their  children  to  be  Liking  full  ad- 
vantage of  all  of  our  programs.  They  also 
want  their  children’s  evaluation  to  reflect 
that  experience.  We  have  an  organization 
in  Toronto  called  the  Organization  of 
Parents  of  Black  Children.  It  is  probably 
one  of  the  most  helpful  groups  we’ve 
dealt  with  and  also  one  of  the  most  chal- 
lenging. They  worked  with  our  staff  to 
produce  a report,  now  several  years  old, 
on  the  education  of  black  students,  and 
one  of  its  key  tenets  and  recommenda- 
tions is  that  we  focus  on  ensuring  that 
children  of  all  groups,  and  particularly 
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We  now  have  very  little 
time  to  assure  people 
about  standards  and  also 
to  provide  some  way  in 
which  our  teachers  can 
talk  with  one  voice  on 
these  issues.  We  began  to 
identify  these  as  early  as 
1984.  The  luxury  of  time 
is  just  not  available  in 
the  same  way  today. 

— Joan  Green 


black  children,  have  the  same  opportuni- 
ties and  that  we  demonstrate  the  same 
results.  Their  fear  is  that  young  people  of 
colour  could  be  clustered  in  one  setting 
where  the  standards  would  not  be  the 
same  as  in  other  settings. 

Russell  What  about  other  groups  in 
the  community,  business  for  example? 
What  is  their  response  to  the  Toronto 
benchmarks? 

Green  Right  now  I’m  working  close- 
ly with  the  Metro  Toronto  Learning  Part- 
nership which  consists  of  directors  of 
education  (three  of  us  serve  on  the  steer- 
ing committee)  and  CEOs  of  major  cor- 
porations in  Ontario,  such  as  Xerox,  Bell 
Ontario,  Northern  Telecom,  Peat  Mar- 
wick Thome.  We’re  working  to  establish 
an  effective  partnership  to  intervene  in 
school  settings  and  help  with  school 
improvement.  Now,  you  can  imagine  the 
politics  of  that.  Many  people  fear  a busi- 
ness take-over,  or  an  imposition  of  a 
business  agenda,  and  therefore  an  ero- 
sion of  the  pure  definition  of  education. 
But  one  of  the  areas  we’ve  been  working 
on  very  hard  is  the  whole  question  of 
what  standards  mean  and  how  business 
views  standards.  We’ve  used  the  Confer- 
ence Board  of  Canada’s  statement  of 
expectations  and  outcomes  as  a way  of 
working  together  and  talking.  So,  there  is 
already  a healthy  dialogue  ongoing  with 
business  leaders  and  educational  leaders 
in  our  jurisdiction  about  this  issue.  We 
really  have  had  some  useful  exchanges 
that  have  helped  people  grasp  the  enor- 
mous complexity  of  human  development 
and  the  range  of  capacities  that  young 
children  have  which  need  to  be  fostered 
as  well  as  monitored. 

Russell  From  your  experience  in  the 
Toronto  Board,  what  are  the  essential 
characteristics  of  benchmarks,  or  those 
which  we  can’t  afford  to  lose? 

Sullivan  First,  it’s  really  important 
that  we  preserve  the  detail  and  clarity  of 
the  benchmark  system,  that  we  keep 
benchmarks  as  models  of  excellence  that 
demonstrate  clearly  to  teachers,  parents, 
and  students  what  it  is  we  are  looking  for 
in  student  work.  Second,  the  actual 
nature  of  the  tasks  we  are  looking  at 
needs  to  be  emphasized,  the  fact  that 
benchmark  tasks  are  much  more  open 
ended  than  multiple-choice  tests  and  pro- 


mote higher  order  thinking  skills.  Bench- 
marks provide  more  than  samples  of  stu- 
dent work;  we’ve  selected  specific 
learning  activities  to  demonstrate  the 
qualities  and  levels  of  performance  we’re 
interested  in  developing  in  all  students. 

Russell  Does  this  attention  to  clarity, 
to  detail,  show  up  in  discussion  at  your 
level,  Joan? 

Green  Absolutely.  If  you  have  a con- 
crete illustration  of  what  a child  is  doing, 
compared  to  a standard  that  has  been  set 
by  the  assessment  of  a whole  range  of 
children  at  that  age,  you’ve  got  some- 
thing very  tangible,  something  parents 
will  be  satisfied  with,  as  opposed  to  feel- 
ing that  they’ve  been  given  mumbo 
jumbo.  The  concreteness  of  the  norms 
we’re  using  provides  parents  with  an 
indication  of  our  willingness  to  be  scruti- 
nized. I think  that  one  of  the  nicest  fea- 
tures of  the  Toronto  benchmarks  is  that 
they  are  so  concrete.  They  are  not  the 
static  school-bound  definitions  that  we 
had  in  the  past  but  they  are  real,  dynam- 
ic, vital  and  allow  and  encourage  lively 
discussion  between  parent  and  teacher. 
When  you  see  your  child  on  video  or 
performing  in  comparison  to  another, 
you  have  a different  kind  of  experience 
than  when  you  read  a piece  of  paper. 

Russell  What  advice,  Marilyn,  do 
you  have  for  teachers  who  are  tempted  to 
begin  using  benchmarks  in  their  class- 
rooms? 

Sullivan  I think  it’s  important  for 
teachers  to  recognize  that  benchmarks 
are  not  outside  the  normal  classroom 
practice,  that,  in  fact,  the  models  illus- 
trated through  benchmarks  are  drawn 
from  normal,  average  classroom  experi- 
ences. So,  the  first  thing  is  to  eliminate 
the  fear  that  this  type  of  assessment  is 
externally  imposed,  or  somehow  differ- 
ent than  what  their  curriculum  has  pro- 
moted for  the  last  decade.  Then  teachers 
need  to  recognize  that  parents  and  stu- 
dents themselves  are  now  looking  for 
much  more  precise  indications  of  the 
quality  of  student  work.  So,  spending  the 
time,  becoming  familiar  with  a bench- 
mark approach  and  starting  to  internalize 
criteria  for  quality  performance,  is  a real- 
ly worthwhile  enterprise. 

Russell  Joan,  what  advice  would  you 
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give  to  CEOs  who  are  thinking  of  start- 
ing up  a benchmark  program  now? 

Green  I don’t  think  there’s  ever  been 
a time,  at  least  in  my  memory  as  an  edu- 
cator, when  we  were  under  such  enor- 
mous scrutiny.  And  in  terms  of  the  fiscal 
restraints  we  face,  a beginning  point  to- 
day would  have  to  be  much  more  accel- 
erated than  ours  was.  We  now  have  very 
little  time  to  assure  people  about  stan- 
dards and  also  to  provide  some  way  in 
which  our  teachers  can  talk  with  one 
voice  on  these  issues.  We  began  to  iden- 
tify these  as  early  as  1984.  The  luxury  of 
time  is  just  not  available  in  the  same  way 
today.  Yet  the  advice  I would  offer  any- 
one who  was  looking  at  this  initiative  is 
based  on  what  I believe  is  part  of  our 
obligation  as  providers  of  educational 
service  to  children,  and  that  is  to  ensure 
they  understand  how  well  they’re  doing 
and  that  their  parents  are  engaged  mean- 
ingfully right  from  the  beginning  as  well. 
We  know  from  all  the  research  that  chil- 
dren will  attain  much,  much  more  suc- 
cess if  their  parents  know  what  it  is 
they’re  doing.  So  I would  make  that  con- 
nection first,  between  accountability  and 
parental  involvement,  and  make  it  very 
clear  we  believe  it  to  be  (a)  our  obliga- 
tion and  (b)  our  pleasure  to  work  with 
parents  because  we  know  that  in  so  doing 
we’ll  be  more  effective  with  the  children. 

Now,  with  respect  to  the  communica- 
tion flow  from  the  CEO’s  office,  I think 
it  is  most  important  to  ensure  that  the 
policy  makers  understand  what  it  is  the 
implementators  are  doing.  It’s  one  thing 
to  have  a board  hear  the  general  aims  of 
a policy  and  then  approve  it.  It’s  another 
thing  to  have  them  understand  it  well 
enough  that  they  can,  among  their  own 
numbers,  count  on  spokespersons  for 
that  policy.  In  our  board,  several  trustees 
have  been  very  outspoken  advocates  of 
our  work,  and  people  like  Fiona  Nelson 
are  called  upon  regularly  to  speak  on 
benchmarks  in  the  media.  So,  first,  you 
have  to  make  sure  your  trustees  arc  with 
you,  and  that  is  a fundamental  task  of  the 
director  of  education. 

I think  the  other  thing  the  director  has 
to  do  on  a very  regular  basis  is  talk  about 
the  advances  and  accomplishments  and 
victories  that  have  been  made  and  tic 
those  to  people’s  work  so  that  they  see  in 
a sense  a picture  of  cohesion.  In  other 
words,  if  any  accountability  scheme  is 
seen  as  an  add-on  or  imposition  and  not 


as  something  integrated  into  the  life  of 
the  school  and  the  priorities  set  for  the 
system  in  terms  of  children’s  learning, 
then  all  that  systemic  institutional  resent- 
ment begins  to  get  in  the  way  and 
emerges  as  yet  another  obstacle.  So, 
there  is  a tremendous  need  to  reinforce 
new  policies  at  every  turn.  I did  a lot  of 
visiting  of  schools  in  my  first  year  as 
director.  I still  teach  two  hours  every 
week.  And  I always  make  a point  in 
working  with  teachers  to  ask  them  to 
give  me  their  latest  insights  on  the  bench- 
mark program.  That’s  actually  how  I stay 
on  top  of  what’s  going  on  in  schools. 

Finally,  I’m  still  of  the  school  that 
believes  principals  are  largely  responsi- 
ble for  the  work  of  staff  and  the  achieve- 
ment of  curriculum  goals.  So  we  need  to 
work  with  our  principals  to  keep  priori- 
ties straight.  It  is  very  important  for  the 
director  of  education  to  be  an  advocate 
of  good  program  policy  and  practice,  not 
just  a person  who  balances  and  crunches 
numbers. 

So,  we  need  the  people  leading  organi- 
zations to  be  involved  in  understanding 
the  partnership  between  parents  and  the 
school  in  terms  of  accountability  for  chil- 
dren’s performance.  We  need  to  keep 
politicians  who  set  policy  in  the  “loop” 
of  understanding  and  we  need  to  keep 
them  informed  about  developments  and 
also  we  have  to  be  candid  with  them 
about  our  failures  or  the  ground  yet  to  be 
broken.  One  of  the  things  I face  all  the 
time  from  politicians  is,  “These  bench- 
marks are  great  theoretically  but  I still 
have  16  parents  calling  a week  saying 
Johnny  is  not  doing  what  Suzie’s  doing.” 
So,  there’s  a tremendous  onus  on  all  of  us 
to  make  sure  that  that  universal  under- 
standing is  there  and  politicians  have  to 
be  told  there  is  still  a struggle.  We  have  to 
devote  whatever  fiscal  resources  we  can 
to  this  activity,  and  reinforce  the  human 
resources  wherever  we  can,  and  we  also 
have  to  be  more  vigorous  and  outspoken 
in  defence  of  curriculum  and  program  pri- 
orities than  we  have  been  in  the  past 
because  as  our  resources  become  serious- 
ly challenged  there  arc  those  who  would 
be  reductionists  and  minimalists.  That’s 
one  of  my  obligations:  to  try  and  keep  the 
system  focussed  and  thinking  in  a “big 
picture”  way. 


The  first 
thing  is  to 
eliminate 
the  fear  that  this  type  of 
assessment  is  externally 
imposed,  or  somehow 
different  than  what  their 
curriculum  has  promoted 
for  the  last  decade. 


— Marilyn  Sullivan 


Russell  Thank  you  for  sharing  your 
experience  with  us.  I’ve  enjoyed  talking 
to  you. 
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Demystifying  the 
Goals  of  Education 

Toronto’s  Benchmark  Program 


Sylvia  Larter 

Research  Officer  (Benchmarks), 

Research  and  Development  Centre, 
Toronto  Board  of  Education 

James  Donnelly 

Principal,  Whitney  Public  School, 
Toronto 

For  more  than  30  years,  the  City  of 
Toronto  chose  not  to  evaluate  its  elemen- 
tary student  population  against  any  inter- 
nal or  external  standards.  Teachers  at 
Toronto’s  more  than  100  elementary 
schools  assessed  students  according  to 
their  own  preferred  methods,  including 
self-made  tests,  cumulative  samples  of 
students’  work,  and  teacher-student  con- 
ferencing. For  reporting  to  parents,  they 
developed  their  own  schemes  — ranging 
in  style  from  highly  personalized  anec- 
dotal reports  to  numbers  and  computer- 
coded  comments. 

Toronto’s  approach  to  evaluation 
afforded  teachers  professional  control 
and  avoided  the  well-documented  short- 
comings of  standardized  tests,  but  it  also 
meant  that  teachers  had  few  moorings  to 
secure  their  judgments  about  student 
achievement.  For  parents  there  were  no 
guarantees  that  teachers  would  not  evalu- 
ate students  either  too  rigorously  or  too 
leniently.  By  the  mid-1980s,  parents’ 
concerns  had  intensified,  and  in  June 
1987,  the  Toronto  Board  of  Education 
passed  several  recommendations  that 
launched  the  Benchmark  Program. 
Included  in  the  recommendations  were 
three  guiding  principles: 

1.  Parents  have  the  right  to  participate 
meaningfully  in  decision  making 
about  their  children’s  education. 


Toronto  defines  a 
benchmark  as  information 
which  teachers,  students, 
and  parents  can  refer 
to  daily  as  they  teach, 
leam,  and  assess 
achievement. 


2.  Parents  have  a corresponding  right  to 
know,  in  a timely  way,  how  well  or 
how  poorly  their  children  are  doing  in 
school. 

3.  Parents  are  entitled  to  have  their  chil- 
dren’s achievements  determined  in 
relation  to  system-wide  standards. 

What  is  a Benchmark? 

Now,  five  years  later,  the  Toronto  Board 
of  Education  has  developed  more  than 
100  Benchmarks  for  Language  and 
Mathematics  at  Grades  3,  6,  and  8 repre- 
senting, respectively,  the  end  of  the  Pri- 
mary, Junior,  and  Senior  Elementary 
Divisions.  In  addition,  work  at  the  Sec- 
ondary Division  is  well  under  way. 

Toronto  defines  a benchmark  as  infor- 
mation which  teachers,  students,  and  par- 
ents can  refer  to  daily  as  they  teach, 
learn,  and  assess  achievement.  Bench- 
marks may  be  shaped  in  any  number  of 
ways,  depending  on  the  needs  of  the 
users,  but  the  standards  they  represent 
are  clear  to  all.  The  philosophy  behind 
the  program  is  that  instruction,  learning, 
and  evaluation  should  occur  simultane- 


ously in  the  classroom  on  a continuous 
basis  with  many  authentic  performance 
activities. 

In  order  to  achieve  this,  teachers  use 
the  benchmarks  as  models  in  designing 
classroom  activities  that  incorporate  the 
Ministry  of  Education  and  Toronto 
Board  of  Education  objectives.  Teachers 
then  observe  students  as  they  perform 
these  activities  in  the  classroom  and 
evaluate  students  according  to  the  levels 
of  performance  depicted  in  the  bench- 
marks. 


Observation  and  Holistic 
Evaluation 

In  developing  the  Language  and  Mathe- 
matics Benchmarks,  the  Toronto  Board 
combined  observation  and  holistic  evalu- 
ation.1 Figure  1 shows  part  of  a Grade  8 
Language  Benchmark.  (The  part  that 
exists  on  videotape  is  not  described.)  At 
the  beginning  are  clusters  of  loosely  con- 
nected, broadly  stated  learning  objectives 
drawn  from  the  Ministry  and  Board 
guidelines.  Next  is  a brief  description  of 
a global  activity,  designed  by  commit- 
tees of  classroom  teachers,  which  opera- 
tionalizes the  objectives.  The  activity 
represents  only  one  of  an  infinite  number 
of  open-ended  tasks  that  could  be 
designed  and  has  no  pre-determined 
answer  or  “correct”  problem-solving 
procedure. 

An  example  of  a Grade  3 Mathemat- 
ics Benchmark  task  (number  16  of  26 
Benchmarks)  is  as  follows: 


Geometric  Solids 

Classifying  Objects  by  Various  Attributes 

Students  sort  a set  of  12  wooden  geomet- 
ric solids  in  two  different  ways.  Where 
necessary,  students  were  given  an  expla- 
nation of  what  is  meant  by  sorting.  In  each 
case,  the  students  were  asked  to  explain 
the  rule  they  used  to  sort  the  solids  into 
sets. 


Both  the  language  and  the  mathemat- 
ics activities  involve  the  use  of  oral  lan- 
guage, and  the  mathematics  activity 
involves  concrete  materials.  Both  tasks 
also  require  interaction  between  the  stu- 
dent and  the  evaluator  — sometimes 
according  to  a structured  format,  some- 
times with  specially  designed  prompts 
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Figure  1 Toronto  Language  Arts  Benchmark  for  Grade  8 


from  the  evaluator,  and  sometimes  spon- 
taneously.2 In  short,  the  activities  resem- 
ble authentic  tasks  that  might  take  place 
in  a regular  classroom  setting. 

Finally,  because  they  are  performance- 
based,  the  activities  require  that  students 
and  their  work  be  observed.  Consequent- 
ly, for  all  such  activities,  the  representa- 
tive samples  of  students  are  videotaped. 


Holistic  Scoring  Criteria 

The  development  of  standards,  of  course, 
suggests  the  need  for  some  form  of  scor- 
ing. Because  the  Language  Benchmark 
task  in  Figure  1 and  in  the  Mathematics 
Benchmark  activity  described  above  had 
been  designed  in  a global  fashion,  and 
because  they  had  never  before  been  con- 
sidered in  the  creation  of  standards, 
developing  criteria  prior  to  the  scoring 
proved  difficult.  It  was  more  beneficial 
to  observe  the  students’  performances  on 


Because  benchmarks 
support  the  developmental 
whole-language 
approach,  teachers  find 
that  the  program  goes 
hand-in-hand  with  their 
classroom  practices. 


the  videotapes  and  simultaneously  deter- 
mine the  criteria.  Thus,  the  standards 
came  to  be  defined  as  what  Toronto  stu- 
dents “can”  do,  not  what  they  “should” 
do.  Because  Toronto’s  student  popula- 
tion is  large  and  diverse  in  its  economic, 
social,  and  cultural  makeup,  these  stan- 
dards provide  sound  and  valuable  infor- 
mation. For  the  type  of  benchmark 
illustrated  here,  we  created  criteria  for 
five  levels  of  performance  so  that  a stan- 
dard shows  the  full  range  of  the  ability  of 
Toronto  students. 

The  holistic  criteria: 

• describe  many  problem-solving  pro- 
cesses, and  higher-order  thinking 
skills; 


KEY  OBJECTIVES 

• Use  oral  language  to  think,  learn,  and  communicate. 

• Read  a painting  and  demonstrate  an  understanding  that  effective 
reading,  of  whatever  type,  is  a constant  search  for  meaning. 

• Demonstrate  aesthetic  appreciation. 


TWO  PAINTINGS  — VIEWING  (ART  BOOK)  AND  RESPONDING  ORALLY 


This  benchmark  was  developed  to  assess 
students'  ability  to  obtain  information  from 
pictures  and  to  explain,  describe,  and  evaluate 
that  information.  Two  paintings,  "The  Fledgling" 
by  E Lindner  and  "The  Young  Canadian"  by 
D.  P.  Brown,  were  chosen  from  the  book  High 
Realism  in  Canada,  by  P.  Duval. 

Students  who  did  well  in  the  activity 
described  and  explained  the  significant 
features  of  the  paintings  and  gave  reasonable 
interpretations  of  them.  They  expressed  their 
ideas  fluently  with  appropriate  language  and 
vocabulary.  They  drew  reasonable  conclusions 
from  the  paintings  and  appreciated  that  their 
ideas  and  feelings  had  value  and  were  worthy 
of  expression. 

Before  the  activity,  students  were  asked 
questions  about  paintings  and  were  rated  for 
prior  knowledge  and  experience.  The 
evaluators  then  showed  the  first  painting  and 


asked  the  students  to  describe  it  as  exactly  as 
possible.  After  responding,  the  students  were 
asked:  (1)  What  does  the  painting  make  you 
think  about?  (2)  How  does  it  make  you  feel?  (3) 
Why  do  you  think  the  artist  did  the  painting  this 
way?  The  evaluators  next  showed  the  second 
painting  to  the  students  and  repeated  the 
same  series  of  questions.  Finally,  both 
paintings  were  shown  together,  and  the 
students  were  asked  to  describe  their 
similarities,  make  up  a title  for  each  painting, 
and  discuss  why  they  preferred  one  painting 
over  the  other.  Upon  completion,  students 
were  asked  to  evaluate  the  activity  in  terms  of 
interest  in  the  paintings,  enjoyment  of  the  task, 
and  ease  of  describing  each  painting. 

The  Grade  6 Benchmark,  L6-2,  shows  how 
Grade  6 students  performed  on  this  same  task 
when  evaluated  with  identical  criteria 


HOLISTIC  SCORING  CRITERIA  USED  FOR  EACH  PAINTING  INDIVIDUALLY, 
BOTH  PAINTINGS,  AND  OVERALL  RESPONSE 

Level  5 The  student  describes  and  explains  the  significant  features  of  the  painting(s)  and 

20%  gives  reasonable  interpretations.  The  student  expresses  opinions  about  the 

painting(s)  and  often  links  them  to  personal  experiences.  The  student  is  able  to 
clarify  assumptions  and  make  inferences  and  can  explain  them  to  the  evaluator 
Views  are  stated  clearly  and  concisely.  The  student  is  articulate  an  self-confident. 
Feelings  are  considered  to  be  valuable  and  worthy  of  expression.  The  student 
explores  ideas  with  the  evaluator  using  appropriate  vocabulary  and 
conversational  style.  The  comparison  of  the  two  paintings  goes  beyond  the 
obvious  and  is  often  thoughtful  and  insightful. 

Level  4 The  student  describes  several  significant  features  of  the  paintmg(s)  with  some 

32%  interpretation.  Some  opinions  about  the  paintings  are  expressed.  The  student's 

description  is  accurate  but  not  figurative.  Views  are  expressed  with  some  fluency, 
and  feelings  are  briefly  stated.  The  comparison  of  the  two  paintings  is  reasonable 

Level  3 The  student  describes  and  explains  one  or  two  features  of  the  paintings  with 

39%  minimal  elaboration.  Opinions  are  rarely  expressed,  and  the  description  may 

include  inaccuracies.  The  student  may  be  hesitant  in  stating  feelings  and  may 
describe  only  a single  similarity  between  the  paintings 

Level  2 The  student  identifies  and  describes  a few  details.  No  opinions  are  expressed.  The 

7%  student  has  difficulty  expressing  feelings,  making  comparisons,  and/or  articulating 

ideas. 

Le ve  1 1 The  student  is  unable  to  respond  or  gt ves  a very  limited  response. 

2% 
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• are  true  to  the  whole  child  performing 
a global  task; 

• reflect  the  complexity  and  intercon- 
nectedness among  the  parts  of  the  per- 
formance; 

• include,  in  an  integrated  fashion, 
many  affective  variables  such  as  per- 
severance, confidence,  and  willing- 
ness, and  some  metacognitive 
variables  such  as  monitoring  one’s 
problem-solving  strategies; 

• and,  because  they  were  drawn  from 
students’  performances,  incorporate 
many  unanticipated,  as  well  as  antici- 
pated, outcomes  and  skills. 

Honouring  Teacher 
Professionalism 

Benchmarks  are  packaged  so  that  the 
standards  can  be  observed.  The  missing 
element  of  Figure  1 is  the  exemplar  of 
students’  performances  on  videotape. 
Other  benchmarks  show  exemplars  of 
students’  performances  in  print  format. 
Teachers  do  not  administer  benchmarks 
to  students  as  they  would  conduct  tradi- 
tional standardized  tests.  Instead,  they 
use  benchmarks  as  reference  materials, 
an  approach  that  has  resulted  in  a variety 
of  centralized  and  school-based  initia- 
tives. 

To  begin,  all  K-8  school  principals 
and  vice-principals  participated  in  one 
week  of  in-service  about  benchmark 
materials  and  philosophy  as  well  as 
strategies  for  school-based  collaborative 
use.  A few  months  later,  school  teams  of 
administrators  and  teachers  participated 
in  centrally  designed  in-services.  Since 
then,  school-based  teams  continue  to  col- 
laborate in  implementing  the  Benchmark 
Program.  The  school  teams  are  using 
funds  from  the  Toronto  Board  and  the 
expertise  of  curriculum  consultants  and 
staff  development  personnel  to  tailor  the 
program  to  their  unique  needs.  Some 
entire  school  staffs  have  gone  off-site 
and  devoted  professional  development 
days  to  the  study  of  benchmarks.  At 
other  locations,  small  groups  of  teachers 
use  preparation  time  and  lunch  hours  to 
share  their  understandings.  Teachers  also 
take  the  benchmarks  home  and  study 
them  on  their  own  time.  The  use  of 
benchmarks  as  reference  material  pro- 


Students  use  benchmarks 
as  models  of  excellent 
performances.  Parents 
consider  them  to  better 
understand  today’s 
complex  menu  of 
educational  objectives 
and  to  make  meaningful 
decisions  about  their 
children’s  education. 


motes  a collaborative  culture,  profession- 
al dialogue,  and  individual  reflection. 


Promoting  Shared  Accountability 

At  Whitney  Public  School  all  teachers 
are  collaborating  to  implement  the  pro- 
gram. Located  in  a wealthy  area  of 
Toronto,  Whitney’s  population  is  476, 
serving  students  from  Junior  Kinder- 
garten through  Grade  6.  Even  though 
benchmarks  have  been  designed  with 
Grades  3,  6,  and  8 students,  the  range  of 
performance  for  each  benchmark  is  so 
wide  that  they  can  be  easily  used  across 
all  grades. 

The  teachers  — organized  in  family 
groups  of  Grades  1/2,  3/4,  and  5/6  — 
agree  on  two  or  three  benchmarks  at  the 
most  appropriate  level  and  work  towards 
a collective  understanding  of  the  tasks, 
criteria,  and  exemplars.  They  then  dis- 
cuss the  skills  students  need  to  perform 
the  tasks,  how  students  learn  the  skills, 
and  how  to  teach  the  skills.  Teachers 
also  look  closely  at  the  role  that  language 
plays  in  students’  demonstration  of  the 
many  higher-order  thinking  skills  associ- 
ated with  each  benchmark. 

Because  benchmarks  support  the  de- 
velopmental whole-language  approach 
(Whitney’s  school  philosophy),  teachers 
find  that  the  program  goes  hand-in-hand 
with  their  classroom  practices.  During  a 
second  phase  of  sharing,  teachers  bring 
to  the  group  samples  of  teaching/leaming 
activities  that  promote  the  development 
of  benchmark  skills  and  discuss  how  to 
integrate  learning  and  evaluation. 


Illuminating  Good  Performances 

This  year  the  Toronto  Board  of  Educa- 
tion is  involving  K-8  teachers  in  design- 
ing a new  system  for  recording  student 
achievement  and  for  communicating 
with  parents.  The  goal  is  to  have  the  sys- 
tem reflect  the  methods  and  philosophy 
of  the  Benchmark  Program  and,  thus, 
create  consistency  across  the  elementary 
and  secondary  levels. 

As  in  many  of  Toronto’s  elementary 
schools,  teachers  at  Whitney  Public 
School  are  experimenting  with  a variety 
of  reporting  forms  and  approaches  to 
teacher-parent  interviews.  They  are  look- 
ing forward  to  presenting  some  of  their 
creative  ideas  to  the  system  later  in  the 
year. 

All  of  Toronto’s  teachers  will  bring  to 
this  exercise  three  years  of  experience 
using  benchmarks  to  evaluate  and  en- 
gage students  and  to  inform  and  engage 
parents.  Unlike  externally  developed  and 
scored  tests,  benchmarks  allow  teachers, 
students,  and  parents  to  collaborate  and 
remain  in  control  of  learning  and  evalu- 
ation. Because  benchmarks  can  be 
observed,  they  demystify  the  goals  of 
education  and  illuminate  the  nature  of 
good  performances.  Students  use  them  as 
models  of  excellent  performances.  Par- 
ents consider  them  to  better  understand 
today’s  complex  menu  of  educational 
objectives  and  to  make  meaningful  deci- 
sions about  their  children’s  education. 


NOTE 

1.  For  an  in-depth  discussion,  see  S.  Larter  (1991), 
Benchmarks:  The  development  of  a new  approach 
to  student  evaluation.  Toronto:  Toronto  Board  of 
Education. 

2.  Because  benchmarks  are  not  to  be  used  as  tests, 
the  scripts  and  scoresheets  designed  to  administer 
the  activities  have  not  been  made  available  to  edu- 
cators. 


Reprinted  from  Educational  Leadership, 
February  1993.  Copyright  by  ASCD. 


For  further  information  about  the  Toron- 
to Benchmark  Program,  contact  Marilyn 
Sullivan,  Superintendent  - Curriculum, 
at  (416)  397-3808  or  Margaret  Evans, 
School  Superintendent,  Toronto  North 
Education  Office  at  (416)  393-0599. 
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Adapting  Benchmarks  to 
French  Immersion  Education 


Working  with  a Grade  3 


Sylvia  J.  Spielman 

Teacher,  Toronto  Board  of  Education 


In  1990/91,  the  Toronto  Board  encour- 
aged all  teachers,  including  French 
Immersion  (FI)  teachers,  to  implement 
the  new  (1989)  Toronto  Board  Bench- 
marks in  their  program.1  Assessment  is 
often  challenging  for  FI  teachers  because 
not  all  assessment  resources  are  designed 
in  French  or  for  the  use  of  FI  students. 
Some  FI  teachers  questioned  whether 
benchmarks  designed  on  the  basis  of 
English  performances  would  be  applica- 
ble in  the  second-language  setting.  I have 
found,  however,  that  many  of  the  FI 
teachers’  concerns  may  be  alleviated. 
My  conclusion  is  based  on  my  use  of  the 
adapted  benchmarks  with  my  FI  students 
and  colleagues,  which  I shall  describe  in 
this  article. 

The  Grade  3 FI  Classroom 

I followed  the  procedure  of  the  Toronto 
Benchmark  Grade  3 Activity  L3-13 
“Reading  a Picture  Book  and  Writing  a 
Story.”  In  preparation  for  the  Benchmark 
activity,  the  students  and  I wrote  a col- 
lective story  on  chart  paper  based  on  a 
novel  that  the  class  had  studied.  The 
children  retold  the  story  in  their  own 
words.  Each  child  contributed  to  the  col- 
lective story.  While  I transcribed  the 
story,  I edited  it  for  correct  vocabulary 
and  syntax  in  an  attempt  to  provide  a 
good  language  model.  The  children 
reread  the  story  and  suggested  improve- 
ments. This  was  followed  by  a choral 
reading  of  the  story  by  the  children  and 
teacher.  When  the  children  were  satisfied 
with  their  story,  I helped  them  put 


Classroom 


together  their  own  criteria  of  what  con- 
stitutes a good  story. 

The  day  following  this  activity,  the 
students  were  asked  to  read  and  write  a 
story  for  Oscar  Fete  Noel.  The  activity 
was  carried  out  similar  to  Benchmark 
L3-13.  The  children  were  instructed  to 
“read”  the  picture  book  and  then  to  retell 
the  story.  They  were  given  as  much  time 
as  they  wanted  but  were  told  that  it  was 
an  entirely  “independent”  activity.  I 


emphasized  that  I wanted  their  best 
story-writing  technique  and  did  a quick 
review  of  their  list  of  characteristics  of  a 
good  story  which  was  on  display.  Stu- 
dents were  reminded  to  reread  and  revise 
their  story. 

School-wide  Holistic 
Scoring  Session 

Copies  of  this  class  set  of  Grade  3 FI 
writing  samples  were  distributed  to  a 
group  of  six  FI  elementary  teachers,  who 
had  just  attended  a workshop  on  general 
holistic  scoring.  The  first  stages  of  the 
evaluation  process  were  done  confiden- 
tially. Each  teacher  was  instructed  to 
quickly  read  the  class  set  of  stories  and 
to  choose  the  strongest  and  weakest  sto- 
ries. The  teachers  then  reread  and  placed 
each  story  in  one  of  five  categories: 
excellent,  very  good,  good,  weak,  or 
very  poor.  Each  story  was  then  evaluated 
using  a number  from  1 to  10.  Here  “10” 


Figure  1 Holistic  Scoring  Criteria  of  L3-13 


Taken  from  Toronto  Board  of  Education  Benchmarks  L3-13 
“ Raking  Lea  ves  With  Max. " 

LEVEL  5 

The  student  demonstrates  a knowledge  of  the  features  of  stories  and  is  aware  of 
the  needs  of  an  audience.  There  is  a complete  telling  of  the  story  with  elaboration 
and  imaginative  details  that  extend  the  information  provided  by  the  pictures.  The 
student  attempts  to  incorporate  feelings  such  as  happiness,  annoyance,  and 
frustration  The  stay  is  cohesive  with  an  effective  opening  and  conclusion  The 
student  may  attempt  to  use  dialogue  and/or  interjections  to  highlight  various 
parts  of  the  story  and  to  engage  the  reader's  attention.  The  writing  shows  control 
over  the  conventions  of  written  standard  English. 

LEVEL  4 

The  student  demonstrates  a sense  of  story  telling  and  some  awareness  of 
audience.  There  is  a complete  and  adequate  telling  of  the  stay  but  with  simple 
straightforward  interpretations  of  the  pictures.  The  story  is  fairly  cohesive  and 
has  an  opening  and  conclusion.  There  may  be  an  attempt  to  use  dialogue.  The 
writing  shows  moderate  control  over  the  conventions  of  written  standard  English. 

LEVEL  3 

The  student  demonstrates  some  control  of  story  fam.  There  is  an  attempt  to  tell 
the  story  and  follow  the  pictures.  Important  details  of  the  stay,  however,  may  be 
omitted.  Sentences  may  be  joined  by  repeated  use  of  "then”  and  there  may  be 
repeated  use  of  certain  phrases.  The  student  may  describe  the  pictures  instead 
of  creating  a stay.  The  writing  shows  some  control  of  the  conventions  of  written 
standard  English. 

LEVEL  2 

The  student  demonstrates  some  control  of  story  fam  There  is  a short  and  simple 
telling  of  the  stay,  sometimes  with  digressions.  Some  of  the  pictures  may  be 
skipped  over.  The  story  may  have  an  appropriate  beginning  but  the  story  telling 
is  not  sustained.  The  student  tends  to  use  short,  incomplete  repetitive  sentence 
patterns.  The  writing  shows  some  control  of  the  conventions  of  wntten  standard 
English. 

LEVEL  1 

No  response  or  very  little  response. 
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represented  the  best  and  “1”  the  weakest 
from  the  entire  group.  Group  discussion 
later  resolved  discrepancies  in  the  teach- 
ers’ ratings.  After  reaching  a consensus, 
we  regrouped  the  writings  into  just  five 
levels.  When  we  discussed  the  character- 
istics of  the  best  — or  level  5 — stories, 


we  discovered  that  many  of  the  teachers’ 
criteria  were  identical  to  those  that  the 
children  had  listed.  The  time  had  now 
come  to  examine  the  holistic  scoring  cri- 
teria (HSC)  developed  for  the  Toronto 
Board  Benchmark  Activity  L3-13  (see 
Figure  1). 


Using  the  Holistic  Scoring  Criteria 

After  closer  examination  of  several  sam- 
ples from  the  class  set,  the  teachers 
agreed  that  many  of  the  Benchmarks’ 
HSC  were  appropriate  for  the  second- 
language  learner.  Many  of  the  character- 
istics of  a good  story  were  similar  in 
both  the  HSC  and  the  teachers’  list.  The 
criteria  were  generic  and  could  be  adapt- 
ed to  assess  other  narratives.  In  addition, 
the  criteria  and  their  corresponding  lev- 
els were  developmental  in  nature  and 
reflected  higher-order  thinking  skills. 

Participants  were  asked  which  criteria 
most  needed  to  be  modified  for  the  sec- 
ond-language writings.  The  criterion, 
“The  writing  shows  (some,  moderate) 
control  over  the  conventions  of  written 
standard  English”  was  often  cited  as  the 
most  problematic.  What  is  an  adequate 
level  of  Grade  3,  written,  standard 
French  for  French  Immersion  students? 
What  are  the  range  of  FI  performances 
across  the  Toronto  Board?  What  are  the 
stages  of  second-language  development? 
The  conventions  of  written  standard 
French  of  the  five  levels  of  Grade  3 
French  Immersion  students  could  not  be 
clearly  defined  without  a lengthy  analy- 
sis and  discussion  using  FI  writing  sam- 
ples (not  included  in  the  original 
Benchmark  package).  The  FI  class  sam- 
ples shared  by  the  teacher  groups  were 
useful  in  describing  the  various  levels  of 
language. 

An  Individual  Assessment 

The  following  individual  assessment 
shows  how  the  benchmarks  translate  into 
the  assessment  and  subsequent  reporting 
to  parents.  Because  of  space  limitations, 
the  written  and  oral  samples  of  only  one 
student  (Student  #18)  are  used. 

Three  FI  teachers  in  attendance  at  the 
school-wide  “Benchmarks  in  FI”  study 
group  agreed  that  the  criteria  of  level  3 
best  described  the  narrative  of  Student 
#18  (Figure  2)  for  the  Benchmark  Activ- 
ity L3-13  (see  Figure  1).  They  identified 
which  criteria  were  present  in  the  child’s 
narrative  and  which  were  not.  They 
compared  the  level  3 and  4 criteria  to 
provide  guidance  in  improving  perfor- 
mance. It  was  noted  that  one  of  die  dif- 
ferences in  the  criteria  was  that  in  level  4 
“there  may  be  an  attempt  to  use  dia- 


Figure  2 Narrative  of  Student  #18 
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Figure  3 Transcript  of  Student  #18’ s Oral  Performance 


Children’s  writing 
samples  can  be  shared 
in  communicating 
effectively  with  parents 
in  report  cards  and 
interviews  and  among 
colleagues  at  school 
team  meetings. 


logue,”  whereas  in  level  3 “the  student 
may  describe  the  pictures  instead  of  cre- 
ating a story.”  The  missing  criterion  of 
“dialogue”  could  become  the  objective 
of  mini-lessons,  (creating  comic  strips, 
role  playing,  etc.)  to  encourage  this  child 
to  “interpret”  (level  4)  the  story  more 
through  dialogue  rather  than  “describ- 
ing” (level  3)  it.  In  addition,  the  teachers 
discussed  the  criterion  “the  writing 
shows  some  control  over  written  stan- 
dard [French].”  Despite  the  language 
weaknesses  identified  in  the  text,  they 
were  also  able  to  point  out  the  strengths 
of  the  child’s  language  use.  Interesting 
words  such  as  “II  bondi”  were  used  (He 
leaped).  The  child  risked  spelling  diffi- 
cult words  such  as  “intusiasement” 
(enthusiastically).  In  addition  to  the 
other  benchmark  criteria  that  character- 
ized the  narrative,  they  commented  on 
the  lack  of  organization  and  legibility 
which  interfered  with  the  comprehension 
of  the  text. 

The  video  Benchmark  L3-5  was  used 
here  to  compare  the  child’s  ability  to 
retell  a story  aloud  as  opposed  to  writing 
a narrative  (L3-13).  A video  was  made 
of  Student  #18  reading  and  retelling  the 
story  Le  Tetard  Mysterieux ; portions 
were  transcribed  to  illustrate  the  child’s 
oral  performance  (Figure  3).  It  must  be 
emphasized  that  the  Benchmark  videos 
were  an  innovative,  rich,  and  unique 
contribution  to  the  domain  of  assess- 
ment. This  transcription  loses  the  “life” 
of  the  retelling.  The  criterion,  “The  stu- 
dent retells  the  narrative  in  a story-telling 
mode  with  a sense  of  fun,  enjoyment  and 
confidence”  is  invisible  to  the  reader  as 


L’histoire  le  T etard  Mysterieux  est  une,  est  une  histoire  avec  un  tetard  qui  ne 
devient  pas  une  grenouille  [child  smiles]  qu’il  a reque  de  son  oncle  pour  son 
anniversaire.  Et  il  apporte  a I'ecole  et  Madame,  urn,  et  le  professeur  lui  dit 
qu'elle  voulait  voir  le  tetard  chaque  jour  pour  que  les  enfants  puissent 
[correct  use  of  subjuctive  verb!]  voir  comment  il  devient  une  grenouille.  Mais 
le  tetard  ne  devient  pas  une  grenouille  [big  smile  and  expression  in  voice]!  Et 
unjour  le  professeur  lui  dit  qu'il  ne  veut  pas  qu’elle,  qu’il  apporte  le  tetard 
encore  parce  que  ce  n'est  plus  un  tetard!  C'est  un  gros  tetard  [says  slowly 
and  with  expression]...  Mais  il-laTete  passe  vite  et  I'annee  scolaire 
recommence.  Et  il  vient  a I'ecole  juste  quand  les  nageurs  ont  plonge  dans 
I’eau  [big  smile].  Aliphonse  est  dedans  il  il  frappe  [student  hits  table  to 
emphasize  “frappe”]  la  cote  la  piscine  et  fait  I'eau  toute  aller  sur  I'entraineur 
[student  laughs  and  is  enjoying  the  mischief  in  the  story],  Les  enfants  courent. 
L'entraineur  est  furieux  avec  Luis.  Luis  est  le  garqon.  Alors  I'entraineur  dit 
qu’il  doive  sortir  Aliphonse  d'ici...  II  achete  le,  le  terrain  de  parking  et  puis  il 
appelle  quelques  constructeurs,  constructures  et  quand  la  piscine  est  finie  il 
invite  tous  les  enfants  de  la  ville.  La  fin. 


Figure  4 Holistic  Scoring  Criteria  ofL3-5 


Taken  from  Toronto  Board  of  Education  Benchmarks  L3-5 
Miscue  Analysis  (Narrative)  and  Oral  Retelling,  7 want  a dog. " 


LEVEL  5 The  student's  retelling  accurately  reflects  the  content  and  the  mood  of  the 
selection.  The  student  retells  the  story  in  sequence,  links  events  and  ideas, 
distinguishes  between  causes  and  effects,  and  makes  inferences.  The  student 
monitors  the  organization  of  material.  With  very  few  prompts,  personal  opinions 
and  reactions  to  the  story  and  its  message  indicate  an  interpretation  of  the  text 
which  goes  beyond  mere  recall.  The  student  strongly  identifies  with  the  story. 
The  language  used  frequently  includes  the  vocabulary  of  the  text,  direct 
quotations  (especially  dialogue),  or  paraphrases  of  the  text  The  student  retells 
the  narrative  in  a story  telling  mode  with  a sense  of  fun,  enjoyment,  and 
confidence.  Language  is  used  effectively  to  recreate  images  of  events. 

LEVEL  4 The  student’s  retelling  reflects  the  content  of  the  selection  The  student  provides 
sufficient  summary  and  detail  to  demonstrate  adequately  an  understanding  of 
the  story.  Information  is  given  in  sequence  and,  with  prompts,  the  student 
elaborates  on  details,  makes  inferences,  and  shows  an  understanding  of  the 
cause  and  effect.  The  student  monitors  the  organization  of  material.  The 
language  is  appropriate  for  retelling  the  story  and  may  include  some  vocabulary 
from  the  text 

LEVEL  3 The  student's  retelling  reflects  the  content  of  the  selection  The  student  may 

provide  a sparse  summary.  Details  and  events  which  are  recalled  may  not  be  in 
sequence.  The  student  requires  much  prompting  and  may  make  inferences  to 
support  some  points  in  the  story.  Occasionally  events  are  linked  to  personal 
experiences.  There  may  be  some  monitoring  of  organization  of  matenaL  The 
language  tends  to  be  less  complex  than  that  of  the  text 

LEVEL  2 The  student's  retelling  contains  a minimal  amount  of  information  It  is 

disorganized  and  disjointed.  Irrelevant  or  inaccurate  information  may  be 
presented.  The  student  requires  much  prompting.  There  is  little  or  no 
self  monitoring.  The  student  tends  to  give  one-word  answers  and  has  difficulty 
expressing  ideas  orally. 

LEVEL  1 The  student  is  unable  to  read  the  text  and  consequently  cannot  attempt  the  task. 
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is  “the  retelling  reflects  the  mood  of  the 
selection.”  I have  occasionally  tried  to 
fill  these  gaps  in  the  transcription  — not 
always  very  successfully. 

I assessed  this  student’s  oral  perfor- 
mance as  being  best  characterized  by  the 
criteria  in  level  5 (see  Figure  4).  In  addi- 
tion, a peer-evaluation  activity  similar  to 
the  previous  listing  of  characteristics  of 
an  excellent  narrative  was  undertaken  to 
generate  a collaborative  list  of  the  peer’s 
oral  retelling  performance.  Again,  the 
students’  list  was  very  similar  to  the  HSC 
of  the  corresponding  Benchmark,  L3-5. 

Sharing  the  video  of  this  child’s  excel- 
lent performance  with  his  peers  seemed 
to  boost  his  confidence.  And  building  on 
the  child’s  strengths  in  oral  storytelling 
could  help  overcome  writing  concerns 
and  certainly  had  many  implications  for 
future  programming. 

Sharing  examples  of  good  writing  by 
peers  proved  meaningful  and  relevant. 
After  samples  were  shared  with  the  stu- 
dent, he  discovered  the  difference  be- 
tween simply  retelling  a story  (level  3 of 
L3-13)  and  creating  one  (as  required  for 
levels  4 and  5).  The  child  was  reminded 
of  trying  to  incorporate  dialogue  where 
appropriate  in  subsequent  writing  activi- 
ties as  one  way  of  working  towards  “cre- 
ating” a story. 

The  classroom  teacher  was  able  to 
illustrate  the  child’s  oral  strengths  and 
writing  weaknesses  concretely  to  the 
child’s  parents.  The  print  and  video  sam- 
ples demonstrated  clearly  the  gaps  in  the 
child’s  abilities.  The  HSC  and  the  levels 
gave  the  parents  a clearer  picture  of  the 
child’s  present  written  and  oral  skills. 


Regular  sharing  and 
collaboration  among 
teachers  should  continue 
to  be  encouraged  to 
identify  the  strengths 
and  possible  adaptations 
of  the  benchmarks  in  the 
French  Immersion  setting. 


The  child’s  strengths  and  concerns  were 
discussed  and  demonstrated  at  the  parent 
interview.  Briefly,  the  child’s  advanced 
oral  language  had  to  be  capitalized  upon 
to  improve  his  writing  skills.  The  child 
was  encouraged  to  have  his  stories  tape- 
recorded  or  transcribed  by  the  teacher  or 
the  francophone  mother.  While  the 
teacher  and  student  worked  on  improv- 
ing fine  motor,  cursive,  written-language 
and  storywriting  skills,  the  child  fre- 
quently used  the  class  and  home  comput- 
er to  combat  temporarily  illegibility, 
spelling  mistakes,  and  slow  rate  of  writ- 
ing. 

Conclusions 

In  my  experience,  the  holistic  scoring 
criteria  provided  a “stable  framework  as 
a reference  point  to  promote  discussion 
among  teachers”  (Don  Rutledge,  speech 
at  OISE,  March  8, 1993).  In  addition,  use 


of  the  HSC  does  not  prevent  teachers 
from  making  any  adaptations,  additions, 
or  deletions  they  deem  necessary.  Sam- 
ples of  authentic  writing  are  useful  and 
interesting  to  use  in  conjunction  with  the 
HSC.  Along  with  the  rich  anecdotal 
information  of  the  HSC,  children’s  writ- 
ing samples  can  be  shared  in  communi- 
cating effectively  with  parents  in  report 
cards  and  interviews  and  among  col- 
leagues at  school  team  meetings. 

Regular  sharing  and  collaboration 
among  teachers  should  continue  to  be 
encouraged  to  identify  the  strengths  and 
possible  adaptations  of  the  benchmarks 
in  the  French  Immersion  setting.  Many 
systems  of  sharing  are  already  in  place: 
the  elaborate  Benchmark  Libraries 
(Teachers’  Information,  Print  and  Video 
student  performances),  BenchPress  (a 
newsletter  for  sharing  ideas  and  strate- 
gies), and  school  and  French  department 
workshops. 

French  Immersion  teachers  need  not 
hesitate  in  implementing  the  Toronto 
Board’s  Benchmarks  in  the  assessment 
of  second-language  skills.  Many  criteria 
of  a good  FI  performance  are  similar  to 
the  criteria  derived  from  English  perfor- 
mances; the  emphasis  on  higher-order 
thinking  skills  seems  to  transfer  across 
the  languages.  And  the  holistic  scoring 
criteria,  based  on  Ministry  and  Toronto 
Board  guidelines,  help  explain  and  iden- 
tify the  objectives  of  the  program. 
Teachers  of  all  grades  can  aim  towards 
these  general  objectives  in  planning 
activities  suitable  for  their  own  students. 


NOTE 

1.  For  more  detailed  documentation  of  the  Toronto 
Board  Benchmark  Program,  see  Larter  and  Spiel- 
man  (1992)  and  Larter  and  Donnelly’s  article  in 
this  issue  of  Orbit. 


REFERENCES 

Day,  A.  (1990).  Oscar  Fete  Noel.  Aartselar,  Bel- 
gium: Editions  Chantecler. 

Kellog,  S.  (1979).  Le  tetard  mysterieux.  Paris:  Edi- 
tions Lotus. 

Larter,  S.  G.,  & Spielman,  J.  S.  (1992).  The  imple- 
mentation of  Toronto  benchmarks  in  the  prima- 
ry/junior divisions.  The  International  Journal 
of  Dynamic  Assessment  and  Instruction,  2(2), 
pp.  68-76. 

Toronto  Board  of  Education.  (1989).  The  bench- 
marks: Grade  three,  six  and  eight  language 
libraries.  Toronto:  Author. 


A subscription  magazine  for  teachers, 
educational  consultants,  and  administrators 
who  want  up-to-date  professional  knowledge 
and  guidelines  for  practice. 

Inexpensive  and  up-to-date.  Comprehensive  coverage  of 
the  latest  in  curriculum  approaches  and  teaching  techniques. 

Four  theme  issues  for  1993.  Only  $30.00  plus  7%  GST  for  a total  of  $32.10 

Violence  in  the  Schools/Schooling  in  Violence. 

Benchmarks,  Standards,  and  Ontario  Curriculum. 

Progressive  Approaches  to  Professional  Development  in  the  ‘90s. 
Changing  Schools/A  Look  at  the  Transition  Years. 


28 


ORBIT 


Christine  Suurtamm 

Mathematics  Teacher 
Peel  Board  of  Education 
Meadowvale  Secondary  School 

A benchmark  is  a standard  by  which 
something  can  be  measured  or  judged; 
it’s  a reference  point.  Let’s  look  at  the 
definitions  of  benchmarks  developed 
recently  by  a number  of  Ontario  boards 
of  education.1 

The  Toronto  Board  of  Education 

The  Toronto  Board  of  Education  began 
its  benchmarks  program  because  they  felt 
it  was  their  responsibility  to  parents  to  be 
able  to  discuss  a child’s  achievement  in 
relation  to  system-wide  standards.  Decid- 
ing just  what  those  standards  would  be 
was  a difficult  task.  The  Board  saw  stan- 
dardized tests  as  evaluating  only  a small 
spectrum  of  curriculum  objectives.  Also, 
they  saw  the  teacher’s  observations  as 
central  to  the  evaluation  of  students. 
How,  then,  could  standards  be  set? 

The  Toronto  benchmarks  are  defined 
by  what  students  are  achieving  and 
reflect  the  “normal”  learning  process. 
They  are  reference  points  for  student 
performance  which  provide  realistic 
goals  and  arc  connected  to  the  objectives 
set  out  in  the  Ontario  Ministry  of  Educa- 
tion’s guidelines.  They  constitute  a holis- 
tic evaluation  since  any  one  task  takes 
into  account  a variety  of  learning  objec- 
tives. Yet  there  arc  very  specific  criteria 
to  guide  the  teacher’s  assessment. 

The  Toronto  Board  has  recognized 
that  it  is  important  to  discuss  the  use  of 


Surveying  the 
Benchmarks 


An  Overview  of  Programs  in 
Five  Ontario  Boards  of  Education 


benchmarks  with  classroom  teachers.  The 
benchmark  library  of  videotapes  and  print 
material  is  fully  accessible  to  teachers. 
The  samples  of  student  work  are  there  to 
aid  teachers  in  their  own  observations 
and  evaluations,  not  as  a set  of  tests  for 
students.  The  samples  give  points  of  ref- 
erence — or  benchmarks  — so  that  the 
teachers  can  assess  where  a student  is  vis 
a vis  certain  learning  objectives  as  com- 
pared to  other  students  in  the  Board. 

Benchmarks  are  also  a way  of  stimulat- 
ing good  teaching  practice.  Teachers  often 
work  in  isolation,  and  using  benchmarks 
to  make  classroom  activities  more  visible 
and  more  discussable  allows  for  enhanced 
professional  growth.  In  some  Toronto 
schools,  teacher  discussion  groups  about 
benchmarks  have  facilitated  an  examina- 
tion of  evaluation  and  observation  prac- 
tices as  well  as  teaching  practices. 

The  Etobicoke  Board  of  Education 

The  Etobicoke  Board  of  Education  has 
been  developing  performance-based 
assessment  techniques  to  ensure  that 
assessment  is  cross-curricular,  reflective 
of  a variety  of  teaching  approaches,  and 
consistent  with  both  Board  and  Provin- 
cial guidelines.  These  Etobicoke  assess- 
ment techniques  also  address  the  broad 
characteristics  of  the  learner  set  by  the 
Board  as  objectives  for  students. 

As  in  the  Toronto  Board,  students 
were  given  tasks,  or  “assessment  tar- 
gets,” which  encompassed  many  curricu- 
lum objectives  and  incorporated  different 
subjects.  They  were  then  evaluated  on 
how  well  they  had  integrated  these  learn- 
ing objectives.  Their  levels  of  achieve- 


ment were  called  “indicators.”  This  Per- 
formance Assessment  Program  shares 
with  the  Toronto  Board  of  Education’s 
Benchmark  Program  the  goal  of  a holis- 
tic evaluation  of  the  student,  but  is  not 
yet  as  highly  developed  or  as  rigorous. 

Frontenac  Board  of  Education 

The  Frontenac  Board  of  Education’s 
benchmarks  program  examined  levels  of 
achievement  in  writing,  a difficult  area  to 
assess.  The  Board  has  undertaken  a Sys- 
tem Review  of  Writing  at  various  grade 
levels.  Each  year,  teachers  at  these  levels 
work  in  groups  to  review  samples  of 
writing  from  students  within  the  Board. 
Using  holistic  scoring  methods,  they 
evaluate  the  students’  writing  samples 
and  place  them  in  a variety  of  categories. 

The  benchmarks  in  this  project  are  the 
rated  exemplars  of  students’  work.  The 
evaluated  student  work  — the  exemplars 
— has  been  catalogued  and  distributed  to 
all  teachers  in  the  Board  who  are  in- 
volved in  evaluating  students’  writing. 
Each  teacher  then  uses  this  catalogue  of 
benchmarks  as  an  aid  in  evaluating  the 
work  of  his  or  her  own  students. 

North  York  Board  of  Education 

The  benchmarks  process  in  North  York 
has  addressed  “learning  outcomes”  for 
reading  and  writing  as  well  as  for  mathe- 
matics. The  process  has  been  cyclical, 
with  constant  testing,  analysing,  and 
revising.  As  outlined  in  The  Benchmark, 
published  by  the  North  York  Board  of 
Education  in  early  1991,  the  project 
began  by  assessing  student  attitudes  and 
achievement  in  board-developed  stan- 
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dardized  tests.  Programs  were  then  ana- 
lysed and  modified  based  on  test  results. 
The  process  involved  determining  meas- 
urable student  “outcomes,”  measuring 
those  outcomes,  reviewing  program 
effectiveness,  and  then  developing  and 
implementing  plans  for  change  or  im- 
provement. 

Teachers  were  encouraged  to  use  a 
variety  of  tools  to  measure  these  out- 
comes. Suggested  methods  included 
teacher  observation,  checklists,  student 
journals,  rating  scales,  self-report  inven- 
tories, and  tests.  Teachers  also  used  the 
results  of  the  board-developed  standard- 
ized tests  to  determine  whether  the  stu- 
dents were  meeting  the  benchmarks 
described  in  the  statements  of  student 
outcomes. 

The  Ottawa  Board  of  Education 

The  Ottawa  Board  of  Education  began 
the  Learning  Profiles  Project  in  1988. 
The  Benchmarks  Committee  was  one  of 
four  committees  on  this  project  and  its 
purpose  was  to  “determine  the  expecta- 
tions for  social/emotional  and  academic 
achievements  at  the  end  of  each  divi- 
sion” (Winks,  1990). 

The  secondary  and  elementary  pan- 
els appeared  to  be  taking  two  different 
approaches.  At  the  secondary  level, 
system-wide  assessment  had  taken  place 
for  quite  some  time.  A common  course 
curriculum  for  all  secondary  level  cours- 
es and  benchmark  “tests”  used  by  the 
Ottawa  Board  had  been  completely  gen- 
erated by  teachers  in  the  Board.  (Perhaps 
this  concept  of  benchmarking  is  more 
closely  related  to  standardization  than  to 
setting  standards.) 

The  Learning  Profiles  Project  was 
more  apparent  at  the  elementary  level. 
Achievement  targets  were  developed,  lev- 
els (or  benchmarks)  of  achievement  for 
those  targets  were  established,  and  exam- 
ples of  student  work  to  illustrate  those 
levels  were  described  or  collected.  Most 
of  the  achievement  targets  were  related  to 
basic  skills  such  as  multiplication  or 
spelling.  Although  achievement  targets 
were  to  be  assessed  within  the  context  of 
other  work,  it  appeared  that  evaluation 
was  focussed  on  those  learning  objectives 
that  were  easiest  to  measure  and  quantify. 
In  comparison  with  the  Toronto  Board, 
Ottawa  appears  to  be  focussing  on  basic 


In  Etobicoke,  learning 
objectives  are  called 
“assessment  targets”;  in 
North  York,  they  are  called 
“learning  outcomes”;  and  in 
Ottawa,  they  are  called 
“achievement  targets.” 


skills  rather  than  evaluating  the  child  in  a 
more  holistic  manner. 

Benchmarking  the  Benchmarks 

Each  board  is  concerned  with  accountabil- 
ity in  terms  of  the  individual  student’s  pro- 
gram as  well  as  the  system-wide  program 
of  education.  In  most  boards,  the  emphasis 
has  been  to  define  what  could  be  expected 
of  students  at  various  grade  levels. 

Another  commonality  is  that  each 
board  has  attempted  to  articulate  a set  of 
learning  objectives  and  levels  at  which 
those  learning  objectives  could  be  met. 
To  some  extent,  each  board  has  devel- 
oped its  own  terminology.  In  Etobicoke, 
learning  objectives  are  called  “assessment 
targets”;  in  North  York,  they  are  called 
“learning  outcomes”;  and  in  Ottawa,  they 
are  called  “achievement  targets.”  The  lev- 
els of  achievement  have  been  called 
“benchmarks”  by  many  boards  but  they 
have  also  been  called  “indicators”  or 
“standard  reference  points.”  The  similari- 


ty in  the  terminology  has  created  a simi- 
larity in  structure  but  within  that  structure 
there  are  many  differences. 

In  examining  the  accountability  of  a 
benchmark  system,  it  is  necessary  to  look 
at  the  articulation  and  choice  of  learning 
objectives  as  well  as  the  forms  of  assess- 
ing whether  those  objectives  are  being 
met.  Learning  objectives  range  from 
acquiring  basic  skills  to  learning  to  inte- 
grate many  disciplines,  to  being  co-oper- 
ative, to  being  a good  problem-solver. 
Each  board  has  taken  a stance  on  what 
should  be  included  in  the  assessment  tar- 
gets or  learning  outcomes.  Some  boards, 
such  as  Toronto  and  Etobicoke,  have 
tried  to  incorporate  a wide  range  of  the 
learning  outcomes  set  by  Board  and  Min- 
istry guidelines.  Others  seem  to  place  an 
emphasis  on  those  learning  outcomes  that 
are  more  easily  measured  by  standard  ref- 
erence points.  In  assessing  a board’s  eval- 
uation policy,  it  is  necessary  to  look  at 
the  range  of  learning  outcomes  for  which 
a board  is  willing  to  be  accountable. 

Another  objective  for  benchmark  sys- 
tems is  the  adequate  assessment  or  evalu- 
ation of  learning  objectives.  There  is 
agreement  among  boards  that  student 
achievement  vis  a vis  learning  outcomes 
cannot  be  assessed  solely  by  standardized 
testing.  But  boards  also  need  to  develop 
teacher  expertise  in  evaluation.  Teachers 
need  guidance  in  developing  assessment 
strategies  for  such  learning  outcomes  as 
recognizing  and  using  patterns,  working 
collaboratively,  and  developing  critical 
thinking.  Some  boards  are  meeting  these 
needs  better  than  others. 


For  more  information  on  the  benchmark  programs  in  these  boards,  contact 


Toronto  Board: 

Marilyn  Sullivan, 

Superintendent  - Curriculum 
(416)  397-3808 

Etobicoke  Board: 

Anne  Berens 

Research  and  Evaluation  Officer 
(416)  394-7321 

Frontenac  Board: 

John  Bates 

Director  of  Education 
(613)  544-6920 


North  York  Board: 

Anne  Gibson 

Superintendent,  Literacy  Outcomes 
(416)395-8296 

Ottawa  Board: 

Ann  Jones 

Assistant  Director  of  Education 
(613)239-2211 
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The  Toronto  Board  has  tried  to  match 
the  assessment  of  individual  students 
with  the  broad  spectrum  of  learning  out- 
comes presented  by  the  Ministry  and 
Board  guidelines  (Larter,  1991).  They  are 
also  attempting  to  help  the  teacher  devel- 
op expertise  in  this  type  of  evaluation. 
The  Frontenac  Board  has  used  samples 
of  holistically  scored  student  writing  to 
help  individual  teachers  in  their  assess- 
ment of  their  own  student’s  writing.  It 
should  be  emphasized  that  a “high  level” 
benchmark  system  also  helps  teachers 
grow  and  develop  professionally  in  the 
area  of  evaluation. 

Each  benchmarking  system  described 
here  is  based  on  a model  of  growth.  The 
assumption  is  that  students  will  all  reach 
the  benchmark  but  they  may  reach  it  at 
different  times.  It  is  a continuum.  And  so 
it  is  with  the  boards’  benchmarking  sys- 


tems. Some  are  in  developmental  stages 
while  others  are  at  the  implementation 
phase.  Just  as  educators  don’t  assess  a 
student  before  having  an  adequate  amount 
of  information,  it  is  probably  unfair  to 
assess  many  of  these  benchmark  systems 
at  this  point.  But  it  is  important  to  listen  to 
the  discussion  and  to  try  to  make  some 
sense  of  it.  Through  dialogue,  collabora- 
tion, and  looking  to  one  another  for  sup- 
port, each  board  across  the  province  has 
been  broadening  its  understanding  of  the 
evaluation  issue  and  benefitting  from 
being  part  of  the  discussion. 

For  further  information,  contact  Chris 
Bridge,  Exec.  Superintendent,  Instruc- 
tional Programs  (416)  890-1099. 


NOTE 

1.  When  this  article  was  researched,  the  Peel  Board 
of  Education  had  just  begun  its  benchmarks  pro- 


ject, “Peel  Learning  Outcomes.”  These  outcome 
statements  and  their  indicators  are  presently  in  the 
final  stage  of  refinement.  For  further  information, 
contact  Chris  Bridge,  Executive  Superintendent, 
Instructional  Programs  (416)  890-1099. 
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One  of  the  major  purposes  of  learning 
mathcmalics  is  to  help  individuals  under- 
stand their  world  and  to  solve  problems 
that  occur  in  it.  For  children  to  develop 
sound  mathematical  understanding  and 
the  ability  to  reason  and  communicate 
mathematically,  they  need  to  be  actively 
involved  in  the  exploration  of  a wide  vari- 


ety of  mathematical  ideas.  This  type  of 
exploration  is  dependent  upon  young  chil- 
dren constructing,  modifying,  and  inte- 
grating ideas  through  interacting  with  the 
real  world,  materials,  and  other  learners. 

Carefully  designed  benchmarks,  as 
part  of  a much  larger  assessment  process, 
can  provide  the  means  for  valuing  the 
thinking  and  understanding  that  children 
develop  in  relation  to  their  mathematical 
ideas.  Benchmarks  can  also  guide  teach- 
ers in  working  towards  the  right  combi- 
nation of  activities,  manipulativcs,  and 
investigations  to  ensure  successful  learn- 
ing experiences. 


School  board  curriculum  guidelines 
are  generally  designed  such  that  the 
broad  Ministry  aim  becomes  a more  spe- 
cific objective  for  a division  or  even,  in 
some  cases,  for  a grade  level.  In  dealing 
with  solid  geometry,  for  example,  the 
board  guideline,  or  intended  curriculum, 
for  the  Waterloo  Region  Roman  Catholic 
Separate  School  Board  contains  a list  of 
specific  instructional  objectives.  One 
objective  is  to  identify  solids  such  as 
cone,  sphere,  cube,  prism,  and  pyramid. 
Specific  instructional  objectives  at  the 
Grade  2 level  include  identifying  and 
naming  these  solids,  relating  them  to 
real-world  objects,  and  sorting  them 
according  to  their  properties. 

In  the  development  of  the  list  of 
board-level  instructional  objectives, 
attention  was  given  to  making  the  objec- 
tives clear  and  to  emphasizing  the  impor- 
tance of  thinking  in  the  manipulation  of 
collectible  and  commercial  geometric 
solids.  Children  were  to  have  opportuni- 
ties to  explain  and  record  their  ideas, 
decisions,  and  experiences.  In  turn, 
teachers  would  be  able  to  analyse  how 
children  think  for  themselves,  choose 
manipulativcs,  make  investigations,  and 
find  solutions  to  mathematical  problems. 
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Implementing  Board  Guidelines 
at  the  School  Level 

Teachers,  however,  may  have  access  to  a 
clearly  defined  list  of  instructional  objec- 
tives for  a specific  grade  level  outlined  in 
a philosophically  sound  curriculum 
guideline.  They  may  also  have  an  ample 
supply  of  appropriate  manipulative  mate- 
rials, and  they  may  have  received  consid- 
erable in-service  with  suggestions  and 
ideas  on  how  to  use  these  materials.  Yet, 
with  all  of  this  assistance,  how  can  we 
ensure  that  a teacher  is  teaching  with  a 


How  can  a teacher  be 
sure  that  the  children 
have  learned  what  is 
expected  of  them? 


problem-solving  focus  in  a specific  area 
of  curriculum?  And  how  can  a teacher  be 
sure  that  the  children  have  learned  what 
is  expected  of  them? 

As  a means  of  addressing  these  con- 
cerns, a small  chunk  of  subject  matter 
and  the  associated  benchmarks  can  be 
examined  closely.  At  our  school,  an  “s” 
list  (or  “school”  list)  has  been  prepared 
for  the  component  of  solid  geometry  at 
the  Grade  2 level.  The  “s”  list  is 
designed  to  clarify  what  is  being  taught 
in  solid  geometry  and  it  contains  a series 
of  anticipated  learning  outcomes,  appro- 
priate and  relevant  to  the  learning  capa- 
bilities of  young  children.  The  items  on 
the  “s”  list  have  been  organized  so  that 
they  match  with  the  objectives  set  for  the 
topic  and  for  maximum  problem-solving 
experience.  The  “s”  list  should  be  the 
taught  or  implemented  curriculum. 

The  “s”  list  for  solid  geometry  — 
which  may  be  considered  a list  of  bench- 
marks — contains  the  following  items: 

• Sort  solids  (many  groups,  two  groups, 
etc.). 

• Identify  individual  sorting  rules. 

• Classify  solids  according  to  a specific 
rule  — for  example,  one  corner,  no 
comers,  more  than  one  comer,  and  so 
on.  (The  child  can  identify  the  classi- 
fication rule.) 


• Select  two  solids  that  are  different  and 
explain  why  they  are  different. 

• Select  a cube , a cylinder,  a sphere, 
and  a cone  from  a collection  of  solids. 

• Use  the  name  of  the  solid  (cone,  cube, 
cylinder,  sphere ). 

• Compare  two  solids  (i.e.,  tell  what  is 
the  same  and  different  about  the 
solids). 

Given  that  the  right  balance  of  experi- 
ences is  provided,  this  “s”  list  should 
help  teachers  identify  those  children  who 
are  performing  at  the  level  consistent 
with  the  benchmark  criteria. 

Start-Up  Activities 

In  choosing  starting-point  activities,  a 
teacher  needs  to  decide  on  the  type  of 
experience  necessary  to  identify  perti- 
nent information  about  the  learner  and  to 
have  some  idea  of  what  information  can 
be  obtained  from  having  a child  do  a spe- 
cific activity.  Some  sample  activities  are: 

a)  Matching  Solids 

Display  an  array  of  collectible  and  com- 
mercial solids  on  a table.  Ask  the  chil- 
dren to: 

1 Find  a solid  like  the  one  in  your  hand. 

2 Choose  two  solids  that  are  the  same. 

3 Choose  two  solids  that  are  different. 

4 Have  the  child  give  some  reasons  for 
his  or  her  choices. 

5 Let  the  children  informally  play  with 
the  materials. 

In  these  small-group  activities,  a 
teacher  is  able  to  observe  and  record  the 
actions  and  verbal  responses  of  the  chil- 
dren. It  is  possible  to  get  some  indication 
of  the  current  level  of  understanding  for 
many  of  the  children  and  of  their  ability 
to  articulate  their  ideas.  Benchmarks  can 
be  used  to  help  the  teacher  focus  on  the 
kind  of  information  to  be  collected  in 
order  to  determine  a starting  level  of  per- 
formance. A teacher  can  gather  informa- 
tion such  as  the  following: 

• the  ability  to  match  objects 

• the  rationale  for  matching  the  same 
objects  (This  may  span  relevant  infor- 
mation — such  as,  “both  can  roll,” 
“both  have  circles  on  the  bottom”  — 


Benchmarks  can  be 
used  to  help  the  teacher 
focus  on  the  kind  of 
information  to  be 
collected  in  order  to 
determine  a starting 
level  of  performance. 


to  irrelevant  information  — such  as, 
“both  have  the  same  colour,”  “both 
are  the  same  size”  — or  no  reason 
may  be  given.  In  some  instances,  a 
child  may  not  be  able  to  choose  two 
solids  that  are  the  same  or  different.) 

• the  rationale  for  matching  different 
objects  (This  is  helpful  to  use  with 
children  who  give  an  outstanding 
response  in  2.) 

* the  level  of  previous  experience  with 
the  materials  (This  could  range  from 
not  knowing  what  to  do  with  the 
materials  to  developing  more  sophisti- 
cated structures.) 

Through  the  use  of  introductory  activ- 
ities, a teacher  can  notice  discrepancies 
or  gaps  between  anticipated  responses, 
actual  responses,  and  benchmark  expec- 
tations. 

b)  Naming  Solids 

Activities  such  as  naming  solids  can  be 
selected  as  a means  of  working  towards 
achieving  a specific  benchmark. 

Place  a set  of  commercial  solids  on  the 
table. 

1 Ask  the  child  to  choose  a solid  that  he 
or  she  can  name  for  you. 

2 Repeat  this  procedure  giving  the  child 
several  opportunities  to  name  other 
solids. 

3 Ask  the  child  which  solid  he  or  she 
thinks  can  roll  the  best.  Which  solid 
would  not  make  a good  roller?  Which 
solid  would  not  stack  well? 

Individual  teacher-child  interactions 
are  important.  It  is  through  the  use  of 
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carefully  selected  tasks  that  a teacher  is 
able  to  develop  a picture  of  what  a 
child’s  performance  tells  about  his  or  her 
level  of  understanding  relative  to  the 
items  on  the  “s”  list.  A teacher  can 
record  the  number  of  solids  that  the  child 
can  correctly  name.  These  responses  can 
be  matched  against  the  names  — cone , 
sphere,  cylinder,  cube.  How  many  of 
these  four  solids  were  named  by  each 
child?  What  was  the  most  frequently 
named  solid?  What  other  solids  were 
named?  The  questions  on  the  property  of 
rolling  can  be  recorded  by  using  a tally 
sheet  for  the  different  solids  selected. 
What  was  the  most  prominent  response 
for  the  best  roller  and  for  the  poorest 
roller?  Is  there  a pattern  in  the  respons- 
es? Often  children  go  into  long  narra- 
tions about  their  choices  and  this  gives 
information  about  the  degree  of  previous 
experience. 

c)  Describing  Solids 

A game  was  played  by  a group  of  four 
children  in  my  class.  One  child  began  by 
selecting  a solid,  hiding  it  behind  his 
back,  and  then  describing  the  shape  for 
the  other  children,  who  in  turn  tried  to 
guess  the  name  of  the  shape.  I recorded 
the  language  used  in  describing  the  solid 
for  additional  insight  into  the  child’s 
understanding  of  the  geometric  concepts. 

Example  #1 
It  is  round. 

It  moves  good,  like  straight. 

It  has  a flat  end  on  each  side. 

Is  it  a cylinder? 

Example  #2 
It  rolls  good. 

It  is  round. 

It  rolls  straight. 

Is  it  a sphere? 

Example  #3 

It  has  a point  at  the  lop. 

It  has  a flat  bottom. 

It  moves  real  funny. 

Is  it  a cone? 

Example  #4 

It  has  sharp  edges. 

It  is  Hat  at  the  lop  and  flat  at  the 
bottom. 

Is  it  a cube? 

Through  examining  the  language 
used,  a teacher  can  analyse  the  degree  of 


clarity  in  the  description,  the  number  and 
nature  of  the  clues  given,  the  type  of  ter- 
minology used,  and  the  level  of  sophisti- 
cation of  the  description. 

Evaluation 

While  there  is  abundant  information  on 
how  to  teach  problem  solving,  there  is 
little  information  on  the  means  of  obtain- 
ing data  and  measuring  the  accomplish- 
ments of  children.  It  is  difficult  to  deter- 
mine the  degree  to  which  the  attained 
curriculum  measures  up  with  the  objec- 
tives of  the  intended  curriculum.  There 
are  no  data  on  how  much  of  the  intended 
curriculum  was  learned  by  how  many 
children,  or  even  which  items  of  the  in- 
tended curriculum  were  not  taught  and 
for  what  reasons. 

In  order  to  obtain  this  kind  of  data,  it 
is  essential  to  keep  ongoing  cumulative 
records  of  the  children  over  time,  as  well 
as  numbers  on  the  amount  of  content  that 
was  known  by  the  children  before  the 


When  dealing  with 
benchmarks,  a description 
of  what  children  can  do, 
because  of  what  was 
taught  to  them,  needs 
to  be  developed. 


teaching  was  started.  For  each  item  on 
the  “s”  list,  a “p”  value  would  be  re- 
quired to  determine  the  “percentage  of 
children”  who  learned  all  the  items  on 
the  “s”  list.  Taught  coverage  could  be 
obtained  also  for  each  item  on  the  “s”  list 
by  counting  the  percentage  of  children 
who  were  taught  the  item. 

Benchmarks 

When  teachers  involve  children  in  the 
types  of  experiences  required  to  teach 
the  “s”  list  items,  they  arc  able  to  gather 
extensive  information  on  their  learning 
activities.  Yet  in  order  to  get  some 
understanding  of  what  is  learned  and 
some  indication  of  how  well  tire  intended 
curriculum  has  been  taught,  it  is  neces- 


sary to  go  through  a process  of  changing 
the  qualitative  data  (teacher  observa- 
tions, children’s  work  samples,  etc.)  into 
quantitative  data  (analytical  scoring).  It 
is  necessary  to  get  scores  on  the  learning. 
For  example,  before  the  teaching,  only 
10  percent  of  the  children  could  do  a 
specific  item  whereas  after  the  teaching, 
60  percent  of  the  children  could  do  the 
same  item. 

When  dealing  with  benchmarks,  a de- 
scription of  what  children  can  do,  be- 
cause of  what  was  taught  to  them,  needs 
to  be  developed.  Then,  in  turn,  clearly 
defined  parameters  or  distinguishing 
attributes  for  each  level  within  a range  of 
levels  need  to  be  identified  for  a specific 
set  of  items  on  the  “s”  list.  In  order  to 
develop  this  set  of  benchmarks,  or  indica- 
tors, it  is  necessary  for  the  teacher  to 
examine  closely  some  videotapes  of  actu- 
al classroom  experiences  and  look  at 
what  the  children  are  doing  on  the  tapes 
relative  to  the  specific  “s”  list  item.  The 
data  are  then  analysed  to  determine 
whether  the  “s”  list  items  have  been 
taught  and  whether  the  objectives  have 
been  attained  and,  more  precisely,  by 
what  percentage  of  the  children. 

In  looking  closely  at  small  segments 
of  classroom  experiences,  it  is  interesting 
to  realize  the  extent  to  which  perspec- 
tives change  and  new  insights  develop. 
Through  analysing  and  reflecting  upon 
the  teaching/leaming  experiences  provid- 
ed for  the  children,  a teacher  can  deter- 
mine possible  directions  for  change  and 
for  personal  growth  (see  Connelly  & 
Clandinin,  1988). 

At  one  time  I perceived  benchmarks 
as  statements  of  learning  developed  by 
some  external  source.  Now,  I feel  that  it 
is  the  teacher  who  is  most  familiar  with 
the  students  and  their  learning  interests 
and  capabilities  and  who  is  in  the  best 
position  to  design  benchmarks  that  en- 
sure the  highest  level  of  student  perfor- 
mance and  the  greatest  enjoyment  of  the 
teach ing/lcaming  process. 
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Linking  Opportunities 
and  Achievement 

A Program  Evaluation  Model 
for  Grade  6 Mathematics 


J.  Stuart  Telfer 

Mathematics  Consultant 
Waterloo  County  Board  of  Education 

The  Review-Development-Implementation 
(RDI)  process  of  implementing  and 
assessing  curriculum  initiatives  allows 
educators  to  think  through  and  plan  each 
stage  of  the  curriculum  process.  However, 
we  spend  so  much  time  fitting  an  educa- 
tional initiative  into  the  RDI  process  that 
it  becomes  of  secondary  importance  to 
consider  if  the  initiative  is  worthwhile 
doing  at  all,  or  if  it  will  result  in  a measur- 
able increase  in  student  learning.  It  is 
essential,  in  my  view,  to  re-focus  our 
attention  on  what  to  change  as  well  as 
how  to  change.  The  what  of  change  must 
be  based  on  analysis  of  the  links  between 
the  learning  opportunities  afforded  stu- 
dents and  the  measures  of  student 
achievement  identified  as  benchmarks  of 
performance. 

I wish  to  explore  the  model  that  has 
formed  the  basis  for  ongoing  evaluation 
of  the  Grade  6 mathematics  curriculum 
in  Waterloo  County.  The  model  has 
benchmarks  that  have  been  identified  by 
County  teachers.  In  fact,  the  strength  of 
the  model  is  that  it  invites,  and  indeed 
requires,  the  active  involvement  of 
teachers  and  mathematics  leaders  in  the 
development,  administration,  and  analy- 
sis of  the  test  items.  This  process  is  a 
positive  step  in  gaining  acceptance  of  the 
evaluation  model. 

The  model  presented  here  is  not  a 
report  card  on  a teacher,  a school,  or  a 
system;  it  is,  rather,  a framework  for 
identifying  and  directing  curriculum 
reform.  It  is  intended  to  give  an  accurate 
and  ongoing  series  of  “snapshots”  of  the 
mathematics  program  in  Waterloo  Coun- 
ty. The  data  collected  are  meant  to 

• provide  direction  for  curriculum 

change  and  teacher  in-service 


• provide  an  ongoing  evaluation  of  the 
mathematics  program  in  order  to 
assess  the  effect  of  future  curriculum 
initiatives 

• provide  information  on  the  degree  to 
which  the  intended  curriculum  is 
being  taught  and  learned 

• identify  those  mathematical  concepts 
that  are  being  taught  but  do  not  signif- 
icandy  improve  student  learning 

• identify  those  mathematical  concepts 
that  are  not  being  taught  by  a signifi- 
cant number  of  teachers  but  should  be 
taught 

• address  the  system  and  community 
needs  for  accountability 

Outline  for  a Model  of  Evaluation 

The  model  is  based  on  that  used  in  the 
Second  International  Mathematics  Study 
(SIMS).  That  model  looked  at  three  cur- 
ricula: the  curriculum  as  prescribed  by 
the  system  (intended),  the  curriculum 
presented  by  the  teacher  (implemented), 
and  the  curriculum  received  by  the  stu- 
dent (attained).  The  comparison  of  the 
data  among  the  three  curricula  gives  vital 
information  about  the  degree  of  confor- 
mity to  the  prescribed  curriculum  and  the 
degree  to  which  teachers  are  successful 
in  presenting  the  curriculum. 

Once  our  benchmarks  are  operational- 
ized, we  have  an  effective  and  relevant 
data-gathering  mechanism  for  defining 
the  intended  curriculum  and  for  measuring 
the  attained  curriculum.  Using  the  SIMS 
model,  we  can  gather  data  about  the 
implemented  curriculum  by  using  both  a 
teacher  and  a student  “Opportunity  to 
Learn”  questionnaire.  Such  questionnaires 
provide  diagnostic  data  to  detect  dis- 
crepancies between  the  students’  attained 
curriculum  and  the  County’s  intended  cur- 
riculum. 

Use  of  this  model  for  program  evalua- 


tion requires  a clear  understanding  of 
what  is  being  evaluated  and  how  the  data 
will  be  used.  Naturally,  as  with  any  type 
of  evaluation,  there  will  be  scepticism 
and  concern  about  the  use  of  the  results. 
Will  the  results  be  used  to  compare 
schools  to  schools;  teachers  to  teachers? 
Teachers  and  administrators  must  be  pre- 
sented with  a clear  statement  of  the 
intended  uses  of  the  data.  By  emphasiz- 
ing the  matrix  sampling  techniques  and 
by  making  benchmark  test  items  freely 
available  to  teachers  well  in  advance  of 
the  testing,  teachers  can  be  encouraged 
to  be  more  receptive  to  the  procedures 
and  more  understanding  of  the  intent  of 
the  testing. 

Large-Scale  Evaluation 
of  Grade  6 Mathematics 

The  Waterloo  County  Mathematics  De- 
partment evaluated  the  Grade  6 mathe- 
matics program  in  preparation  for  revi- 
sion to  its  K-6  mathematics  curriculum 
guidelines.  From  September  1989  to 
April  1990,  the  Department  — with  the 
assistance  of  Dr.  Merle  Wahlstom  of 
OISE  — developed  a plan  for  the  pro- 
ject, created  the  benchmarks,  and  de- 
vised the  data-collection  mechanisms. 
The  review  focussed  on: 

• student  achievement  data  for  selected 
benchmarks  in  a balanced  mathemat- 
ics program  in  order  to  obtain  mea- 
sures of  the  attained  curriculum 

• student  opportunity-to-leam  data  for 
selected  benchmarks  in  a balanced 
mathematics  program  in  order  to 
obtain  measures  of  the  taught  curricu- 
lum 

• student  and  teacher  attitude  informa- 
tion related  to  the  current  mathemat- 
ics program 

Sample  “Opportunity  To  Learn” 
(OTL)  Question: 

Objective  D* 

Which  of  these  represents  eight  hundred 
and  seven  and  three  tenths? 

A)  807.3  C)  870.3 

B)  807.03  D)  873 

1.  During  this  school  year,  did  you  teach 
or  review  the  mathematics  concepts  or 
skills  needed  to  answer  the  item  correct- 
ly? 

no  . . . yes  . . . 
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2.  If  yes,  please  indicate  in  which  term 
the  major  portion  of  this  instruction 
occurred  (select  1 only). 

...  Term  1,  Sept.-Dee. 

...  Term  2,  Jan -Mar. 

...  Term  3,  Apr. -June 

3.  If  no,  the  reason  it  was  not  taught  was 
(select  1 only): 

...It  will  be  taught  later  in  the  school 
year. 

...  Students  were  not  developmental^ 
ready  to  learn  the  concepts. 

...  Students  already  understood  the  con- 
cepts. 

. . . The  lack  of  time  to  teach  the  concept 
in  the  program. 

...  Other  topics  took  priority. 

Question  #1  is  used  to  calculate  the 
teacher  OTL  index  for  each  objective. 
Question  #3  allows  clearer  study  of  why 
certain  intended  objectives  were  not 
taught,  and  Question  #2  will  be  used  to 
find  if  an  increase  or  decrease  in  p-val- 
ues  (percentage  correct)  can  be  related  to 
the  time  of  year  instruction  is  given.  For 
example,  most  teachers  teach  Grade  6 
geometry  in  May/June.  A high  p- value 
may  be  related  to  the  close  proximity  of 
the  concept  to  the  test  time  rather  than  to 
improved  student  understanding.  It  may 
also  be  found  that  a particular  topic  is 
taught  by  most  teachers  in  January/ 
February.  A significant  reason  for  a low 
p-value  may  then  be  related  to  the  time 
of  instruction  — that  is,  winter  doldrums 
— rather  than  the  difficulty  of  the  topic. 

The  design  of  the  model  offers  both  a 
pre-test  and  a post-test  and  time-series 
analysis  of  the  data.  When  looking  at  the 
data  from  both  design  procedures,  the 
central  issues  of  causality  and  generaliz- 
ability  must  be  discussed.  The  design 
provides  annual  data  relating  student  pre- 
test and  post-test  p-values  for  each 
objective  as  well  as  a c-value  coverage 
index  (the  percentage  of  students  taught). 
These  data  can  provide  valuable  infor- 
mation to  identify  areas  for  which  teach- 
er in-service  is  needed. 

Despite  my  belief  that  the  design  will 
produce  valuable  program  evaluation  data 
each  year,  the  inherent  weaknesses  of  a 
pre-test,  post-test  design  over  a onc-ycar 
period  require  that  we  look  at  this  process 
as  part  of  an  annual  program  evaluation 


strategy  that  will  yield  comparative  data 
from  year  to  year.  The  design  offers 
results  that  are  statistically  sound  and 
generalizable.  The  pre-test  should  have 
little  or  no  influence  on  the  success  of  the 
student  writing  the  post-test  and  should 
certainly  be  a negligible  factor  from  year 
to  year.  The  selection  process,  being  ran- 
dom, will  not  jeopardize  the  external 
validity.  As  well,  since  the  respondents 
change  from  year  to  year,  there  will  be  no 
multiple  treatment  interference. 

By  looking  at  the  post-test  data  from 
year  to  year,  we  can  also  begin  to  get  a 
picture  of  an  evolving  (rather  than  re- 
volving) mathematics  program.  I will 
examine  two  possible  scenarios.  By 
compiling  annual  data  each  year,  the 
Waterloo  mathematics  community  can 
be  aware  of  trends  in  student  understand- 
ing of  specific  mathematics  benchmarks. 

Objective  #5:  Dividing  Fractions 


Scenario  1 


Scenario  2 


In  the  plot  shown  in  Scenario  1,  there  is 
a clear  indication  that  student  skill  in 
dividing  fractions  is  deteriorating.  There 
is  also  an  indication  that  the  cause  may 
be  directly  correlated  to  the  students’ 
opportunity  to  learn  the  skill.  Is  it  impor- 
tant for  students  to  know  how  to  divide 
fractions?  If,  in  1997,  it  is  not  identified 
as  an  important  skill,  we  may  take  the 
initiative  to  reduce  the  OTL  further.  If  it 
is  deemed  important,  we  will  need  to  ask 


whether  increasing  OTL  will  improve 
student  performance  and  by  how  much. 
The  data  can  alert  educators  to  the  situa- 
tion and,  possibly,  to  a resolution. 

In  the  plot  shown  in  Scenario  2,  the 
OTL  has  remained  fairly  constant,  but 
there  is  an  indication  of  a significant  p- 
value  increase  during  1992/93.  What  is 
the  cause  for  the  increase?  Does  this 
indicate  the  success  of  a particular  cur- 
riculum initiative  or  professional  devel- 
opment activity?  Why  has  the  p-value 
increased  fairly  consistently  while  the 
OTL  has  remained  constant? 

What  these  two  scenarios  exemplify 
is  that  on-going  program  evaluation  may 
not  only  provide  answers  to  many  Ques- 
tions, but,  just  as  important,  it  will  also 
provide  the  rationale  and  the  justification 
for  these  questions  — questions  that  are 
based  on  improving  student  learning  and 
identifying  concepts  worthy  of  student 
and  teacher  time  to  learn. 

Summary 

What  I have  attempted  to  describe  is  an 
active  and  viable  process  for  evaluating 
the  mathematics  curriculum  in  Waterloo 
County  based  on  the  creation  of  bench- 
marks. The  central  theme  of  my  article  is 
that  the  mathematics  community  simply 
does  not  possess  the  data  needed  to  direct 
purposefully  improvements  in  the  mathe- 
matics curriculum.  We  all  have  biased 
perceptions  and  we  can  all  speculate,  but 
we  can  no  longer  direct  our  programs 
based  on  these  speculations.  The  model 
and  the  procedures  I have  described  can- 
not and  will  not  provide  all  the  data  neces- 
sary to  direct  accurately  curriculum 
change.  It  is  essential,  however,  for  educa- 
tors to  begin  the  process  and  to  begin  with 
a model  that  looks  at  more  than  just  stu- 
dent achievement  scores. 

The  SIMS  model  allows  an  analysis 
of  the  key  components  in  the  learning 
process:  what  should  be  taught,  what  is 
taught,  and  what  is  learned.  The  model 
allows  for  the  establishment  of  links,  bet- 
ter learning  opportunities,  and  bench- 
marks. A program  evaluation  designed 
with  those  parameters  can  provide  teach- 
ers and  administrators  with  a description 
of  the  program  dial  is  directly  related  to 
improving  the  classroom  experience,  not 
just  to  providing  a scorecard  on  perfor- 
mance. 


* These  “objectives”  can  be  considered 
benchmarks  with  a little  license. 
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A DISCUSSION 


“How  well  are 
we  doing?” 


A Discussion  with  Bruce  Archer,  Burle  Summers, 
Ken  Travers,  and  Howard  Russell 


Bruce  Archer 

Executive  Assistant, 

Ontario  Teachers’  Federation 

Burle  Summers 

Director,  Learning  Assessment  Branch 
and  Project  Manager,  Provincial 
Standards  (Benchmarks)  Project, 
Ontario  Ministry  of  Education 
and  Training 

Ken  Travers 

Professor  of  Mathematics  Education 
University  of  Illinois  at 
Urbana-Champaign 

formerly  Director,  Division  for  Research 
Evaluation  and  Dissemination 
National  Science  Foundation, 
Washington,  DC 

Moderated  by  Howard  Russell 


Russell  Ken,  would  you  comment  on 
“standards”  and  “benchmarks”  as  you 
see  them  from  your  position  at  the 
National  Science  Foundation? 

Travers  First,  we’re  very  concerned 
here,  as  others  elsewhere  are,  with  the 
quality  of  science  and  math  that’s  avail- 
able to  our  students.  We  use  words  like 
“world-class  standards”  for  curriculum. 
And  it’s  very  helpful  for  us  to  think 
about  issues  in  the  international  context. 
From  my  experience  with  the  Second 
International  Mathematics  Study,  where 
we  collected  data  ten  or  so  years  ago,  we 
got  a lot  of  mileage  out  of  distinguishing 
three  notions,  or  levels,  of  curriculum. 
One  was  to  think  about  what  the  coun- 
tries expect  students  to  know  in  science 
and  mathematics  — that  is,  the  intended 
curriculum.  Currently,  we’re  very  inter- 
ested in  seeing  what  the  various  educa- 
tional systems  expect  learners  to  know  at 
various  age  levels.  And  in  the  Third 
International  Mathematics  and  Science 
Study,  we’re  also  looking  at  exit  exami- 
nations around  the  world,  for  students 
leaving  high  school  and  going  to  college. 
What  are  the  sets  of  expectations  that 
countries  have  for  students  at  that  point? 
Yet  as  we  all  know,  there’s  a big  differ- 
ence between  what  is  in  the  curriculum 
guide  (the  intended  curriculum)  and  what 
ends  up  in  the  classroom  (the  implement- 
ed or  delivered  curriculum).  We  have 
come  up  with  various  ways  of  getting  a 
handle  on  the  implemented  curriculum. 
There’s  been  a lot  of  creative  work  done 
on  this  by  folks  at  OISE,  particularly 


Richard  Wolfe,  who  devised  a novel  way 
of  displaying  data  that  plots  classroom 
data  against  data  on  the  intended  curricu- 
lum (opportunity  to  learn).  These  plots 
present  dramatic  evidence  that  class- 
rooms differ  in  terms  of  curriculum  top- 
ics that  are  covered.  By  looking  at  those 
data,  on  what’s  actually  taught  to  stu- 
dents, we’re  better  able  to  get  a handle 
on  the  third  level  of  curriculum  — that 
is,  student  achievement. 

Russell  Does  the  word  “standard,” 
then,  push  benchmarks  aside  in  your 
world?  Or  are  both  words  going  to  be 
around  for  a while? 

Travers  We’re  just  now  getting  into 
the  notion  of  standards  in  the  more 
empirical  sense  and  I’m  sure  we’ll  be 
struggling  with  that  for  quite  a while. 

Russell  Burle,  can  I ask  for  an  update 
from  the  Ontario  Ministry  of  Education 
and  Training?  It  sounds  like  you’re 
wrestling  with  the  same  problem. 

Summers  The  question  of  standards 
has  preoccupied  us  for  many  years. 
We’ve  been  very  soft,  I think  generally, 
on  how  we’ve  articulated  standards.  But 
the  current  demand  is  for  us  to  describe 
them  more  clearly  and  more  publicly, 
more  universally,  and  to  apply  them 
more  consistently.  We’re  also  struggling 
with  an  “outcomes”  orientation  to  the 
question,  “How  well  are  we  doing?”  Ken 
quite  rightly  pointed  to  the  fact  that 
people  expect  us  to  go  beyond  our  local 
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jurisdiction,  in  our  case  the  province.  So 
the  next  question,  once  we’ve  articulated 
the  standard  and  examined  the  outcome 
against  it,  is  “How  does  it  compare  with 
Japan  or  Germany  or  Britain?”  There’s  a 
global  aspect  to  the  dialogue.  The  pro- 
cess of  having  public  communication 
on  both  the  outcomes  or  results  to  be 
achieved  and  the  standards  to  be  applied 
to  them,  and  getting  some  agreement,  is 
the  current  challenge. 

Russell  Bruce,  I wanted  to  get  your 
reaction  to  this  discussion  from  the  per- 
spective of  a teachers’  organization? 

Archer  I back  up  to  the  concept  of 
accountability.  At  one  level,  people  just 
want  their  tax  dollars  to  buy  quality  edu- 
cation. They  want  to  look  at  a current, 
relevant  education  system.  In  particular, 
parents  want  to  know  how  their  kids  are 
doing,  whether  they’re  progressing  from 
earlier  stages,  relative  to  other  kids,  and 
meeting  some  understandable  set  of  stan- 
dards. It  may  be  that  the  principal  ingre- 
dient in  all  of  this  is  to  tie  the  public  — 
and  that  includes  parents  and  the  busi- 
ness world  — into  a more  open,  collabo- 
rative system  of  education.  That  involve- 
ment will  provide,  I believe,  the  assur- 
ances that  we  are  evolving  standards. 

Russell  What  do  people  want  in 
terms  of  accountability? 

Travers  We  tend  to  be  intimidated 
when  folks  say  that  the  government  — 
that  is,  the  people  who  send  us  money  to 
do  things  — wants  evidence  in  terms  of 
student  achievement  scores  that  our  pro- 
grams are  really  making  a difference.  It’s 
not  that  achievement  scores  are  unimpor- 
tant — they  certainly  are  — but  we’d 
like  to  have  a lot  of  information.  First  of 
all,  we  want  information  about  what  the 
students  are  like,  what  the  array  of  learn- 
ing is  that  takes  place,  not  only  in 
achievement,  but  in  attitudes  and  in  ways 
of  thinking  about  science.  I mentioned 
earlier  the  notion  of  the  delivered  cur- 
riculum. We  want  a better  handle  on  test- 
ing in  terms  of  equity.  One  of  our  con- 
cerns is  to  what  extent  arc  the  innova- 
tions that  we  fund  promoting  equity  in 
terms  of  access  to  quality  science  or 
math  education.  So  we  like  to  think  here 
that  our  clients  arc  in  the  market  for  a 


broad  array  of  indicators  and  not  simply 
test  results.  Maybe  part  of  our  job  is 
social  marketing  in  that  we  need  to 
create  an  interest  out  there  in  the  broader 
view  of  education. 

Summers  Can  I pick  up  on  the  notion 
of  a “new  collaboration,”  which  Bruce 
alluded  to  earlier?  As  he  said,  we  have  to 
involve  the  public  in  education.  We’re 
hearing  that  from  a variety  of  sources. 
But  pulling  that  dialogue  off  requires 
sharing  our  turf,  talking  to  people  in  an 
open  setting  where  they’re  comfortable, 
not  just  working  in  the  confines  of  a 
classroom.  A major  obstacle  we  have 
here  is  with  the  language  we  use.  We  are 
constantly  challenged  to  talk  more  simply 
about  these  concepts  so  that  people  can 
participate  in  the  debate.  First,  those  of  us 
in  education  have  to  collectively  agree  on 
our  language  and  then  use  it  consistently. 
Another  obstacle  is  that  the  public’s  expe- 
rience has  been  with  scores,  grades, 
marks,  and  so  forth.  So  there’s  a great 
need  for  public  education,  as  Ken  sug- 
gested, about  the  other  indicators  or  the 
pieces  of  evidence  — and  “evidence”  is 
the  word  I’m  using  quite  consistently  now 
— which  indicate  achievement.  What  evi- 
dence would  demonstrate  that  a student 
has  learned?  What  evidence  do  parents 
have  that  skills  have  been  acquired? 

Travers  When  the  results  of  the  Sec- 
ond International  Math  Study  came  out, 
we  were  under  a lot  of  pressure  from  the 
media  to  rank  how  we  stood  in  compari- 
son to  other  countries.  We  resisted  that  a 
great  deal  but  were  not  totally  successful. 
We  insisted  in  our  report  on  presenting 
information  on  sub-score  level  algebra 
and  geometry  and  where  at  all  possible  at 
the  item  level.  We  had  a major  story  on 
“Good  Morning,  America.”  What  they 
did,  and  it  was  so  nice  I thought,  was 
they  gave  an  example  of  an  item  from 
our  test,  and  used  that  to  explain  the 
standing  of  American  kids,  in  the  context 
of  other  countries. 

Russell  I think  it’s  important,  too,  to 
be  that  concrete  with  the  generation  of 
our  data.  That  really  takes  it  beyond  the 
testimonial  and  rhetorical. 

Travers  And  now,  with  all  the  tech- 
nology we  have  for  making  very  impres- 


sive data  displays,  we  can  be  much  more 
creative  in  how  we  use  the  various  media 
out  there  to  tell  our  stories. 

Russell  Should  we  introduce  the  con- 
cept of  “high-stakes  testing”  at  this 
point?  It  does  seem  to  have  impacted  on 
the  discussion,  on  Burle’s  work  at  the 
Ministry,  and  I know  that  Bruce  has 
thought  long  and  hard  about  it  because  it 
has  impacted  on  the  lives  of  teachers  at 
the  classroom  level. 

Archer  High-stakes  testing  is  ah  old 
concern  for  practising  professionals  who 
worry  that  somehow  all  the  evidence 


We  all  know, 
there’s  a big 
difference 
between  what  is  in  the 
curriculum  guide  and 
what  ends  up  in  the 
classroom. 

— Ken  Travers 


derived  from  testing  will  skew  educa- 
tion, will  have  negative  results  in  all 
sorts  of  ways,  in  terms  of  promotion  and 
tenure  and  adverse  judgments. 

Summers  The  high-stakes  aspect  of 
testing,  which  manifests  itself  in  state- 
ments comparing  students  or  teachers  or 
schools,  is  very  much  a concern  of  the 
public  at  the  moment.  Even  some  of  our 
politicians  talk  openly  about  their  con- 
cerns for  ranking  students  or  schools. 
What  we’ve  tried  to  do  is  help  them 
understand  that  testing  for  ranking  isn’t 
helpful  in  achieving  improvement,  dial  it 
isn’t  motivating,  and  it  doesn’t  have  the 
desired  effect.  It’s  all  very  well  to  state 
that,  we  must  prove  it.  1 believe  we  have 
to  report  ranges  of  scores  and  look  at  sub- 
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scores  and  do  item  analyses,  as  opposed  industry  and  the  enormous  inertia  not 

to  the  gross  aggregrations  and  averages  only  in  terms  of  the  test  companies,  but 

that  we’ve  tended  to  use  in  the  past,  and  also  the  materials  in  textbooks.  I’m 

then  invite  people  to  look  at  very  con-  pained  to  say  that,  but  the  publishers 

Crete  areas  of  knowledge  and  skills  where  have  enormous  control  over  the  nature  of 

we  have  high  achievement  or  where  we  instructional  experience  available  to 

_ teachers  and  students. 


The  high- 
stakes  aspect 
of  testing, 
which  manifests  itself 
in  statements  comparing 
students  or  teachers  or 
schools,  is  very  much 
a concern  of  the  public 
at  the  moment. 

— Burle  Summers 


need  to  improve.  The  public  is  smart 
enough  to  see  the  value  of  such  analysis 
and  to  recognize,  on  a personal  level,  that 
simply  ranking  individuals  and  publish- 
ing the  results  either  for  students  or 
schools  does  not  have  the  desired  effect. 

Travers  I think  what  we  really  need 
to  do  is  address  the  nature  of  our  assess- 
ment activities.  One  of  the  most  visible 
studies  we  funded  here  in  the  last  while 
came  up  with  the  remarkable  finding  that 
our  instruments  emphasized  low-level 
skills  — recognition,  recall,  and  so  on  — 
with  very  little  emphasis  on  subject  mat- 
ter and  the  context  of  what  we’re  trying 
to  accomplish  here  at  the  National  Sci- 
ence Foundation.  That  is,  the  goals  for 
assessment  were  just  totally  counter  to 
what  we  value  in  science  and  math  edu- 
cation. So,  we’re  putting  a lot  of  effort 
now  into  alternative  modes  of  assess- 
ment that  emphasize  the  curricular  goals 
we  want  to  promote  here.  One  of  the 
major  barriers  right  now  is  the  testing 


Summers  Just  as  Ken  indicated,  we 
tend  to  talk  about  very  high-order  goals 
related  to  problem  solving,  but  we  test  at 
an  extremely  low  level.  That  discrepancy 
is  noted  not  just  by  educators  but  by  the 
public  and  certainly  by  the  business 
community.  So,  we  must  generate  better 
assessment  techniques.  An  accountant  in 
charge  of  training  accountants  in 
Ontario,  who  had  reviewed  the  math 
benchmarks,  put  it  this  way:  “I  wouldn’t 
presume  to  tell  you  what  to  teach  in 
Grades  3,  6,  and  9,  but  I really  like  your 
problem  orientation.  We  need  to  do  that 
better  in  training  our  accountants  so 
they’re  no  longer  simply  bean  counters 
but,  in  fact,  members  of  a team  that  con- 
tributes to  solving  major  complex  prob- 
lems in  an  organization.  It’s  all  very  well 
to  have  that  goal,  but  doing  it,  training 
people  to  do  that,  and  assessing  it,  is  the 
major  challenge.”  I was  pleased  to  get 
that  kind  of  support  from  outside  educa- 
tion for  what  we’ve  been  trying  to  do  in 
assessment  and  evaluation. 

Travers  In  some  of  the  samples  of 
assessment  activities  that  we  provided  in 
math,  we  consciously  avoided  simple 
algorithms  because  we  didn’t  want  to 
give  the  impression  that  that  is  all  there 
is  to  mathematics.  We  wanted  to  be 
working  at  a higher  level.  We  had  been 
told  by  the  organizations  that  we  were 
not  going  far  enough  in  terms  of  higher- 
order  thinking  skills  and  problem  solv- 
ing. The  feedback  we  got  universally  was 
that  there  was  no  evidence  of  arithmetic 
in  what  we  were  doing.  As  soon  as  we 
went  back  and  built  in  arithmetic  that  they 
recognized  and  understood,  then  they 
were  quite  prepared  to  look  at  higher- 
order  thinking  skills  and  problem  solving 
in  mathematics  that  related  to  science  and 
technology  in  a way  we’d  all  approve.  So, 
I guess,  linking  what  we’re  trying  to 
achieve  to  what  people  know  and  under- 
stand is  still  a critical  factor  in  terms  of 
implementation. 


Russell  Bruce,  what  about  this  direc- 
tion? It  seems  both  Ken  and  Burle  are 
talking  about  evaluating  higher-order 
complexity  and  a different  mechanism 
for  gathering  the  data. 

Archer  I think  all  of  these  comments 
can  be  used  to  make  one  point,  namely 
that  a principal  reason  for  investigative 
research  in  testing  and  other  forms  of 
measurement  is  to  improve  the  overall 
process  and  our  understanding  of  it. 
Maybe  we  can  never  avoid  invidious 
comparisons  of  teacher  to  teacher,  board 
to  board,  jurisdiction  to  jurisdiction,  but 
the  right  kinds  of  explanations  about 
what  testing  can  do  can,  I think,  mini- 
mize the  adverse  effects.  I suppose  what 
I’m  coming  to  is  that  testing  ought  to  be 
constantly  asking  itself,  “What  is  it  we’re 
trying  to  do?”  And  what  we’re  not  trying 
to  do,  in  my  view,  is  simply  provide 
graphical  information  because  that’s 
open  to  misconstruction,  especially  in 
the  hands  of  media.  So,  I’m  very  much 
in  favour  of  what  I’m  hearing. 

Russel!  And  are  you  in  favour  of  the 
direction  that  benchmarks  seem  to  be 
leading  us? 

Archer  Well,  I think  benchmarks  pro- 
vide a linkage  between  the  three  stages 
of  curriculum  — intended,  implemented, 
and  attained.  It  makes  them  work  with 
each  other  in  a mutual  and  complemen- 
tary way. 

Russell  Do  you  see  benchmarks  in 
the  same  way,  Ken? 

Travers  Yes,  I do.  And  I reiterate  the 
importance  of  the  exemplars,  the  instruc- 
tional goals  we  really  want  to  achieve. 

Russell  And  Burle? 

Summers  I’d  like  to  comment  by 
way  of  paraphrasing  what  a woman  in 
Northern  Ontario,  speaking  on  behalf  of 
Canada’s  First  Nations,  said  to  me  in 
response  to  the  questions  about  stan- 
dards. Should  there  be  standards?  How 
should  we  go  about  setting  them?  How 
should  they  be  used?  Her  comment  was 
that  standards  are  not  the  problem.  It’s 
what  we  do  with  them  in  schools  that’s 
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from  what  students  can 
do  in  a certain  context, 
as  opposed  to  what 
they  should  be  doing. 

— Bruce  Archer 


the  problem  — we  continue  to  use  them 
to  simply  fail  people  or  to  mark  how 
deficient  they  are  and  make  news  stories. 
Frequently  people  know  they  don’t  meet 
the  standard  and  that’s  why  they  go  to 
school  — to  improve,  to  learn,  to  grow. 
She  said  that  if  we  would  use  the  stan- 
dards to  point  the  direction,  to  give  clear 
signposts,  and  generate  indicators  of  suc- 
cess, and  then  help  people  move  along 
that  pathway  to  a goal,  that’s  what  stan- 
dards are  for.  She  asked  us  to  please  get 
it  right.  I thought  it  was  a wonderful 
challenge  for  those  of  us  who  are  devel- 
oping benchmarks  for  Ontario,  what  we 
now  are  calling  provincial  standards  in 
mathematics  and  language. 

Russell  Can  I move  you  now  to  define 
the  most  important  features  of  bench- 
marks? 

Summers  I view  benchmarks  as  a 
standards-setting  process  and  activity.  It 
engages  the  profession  in  being  clear 
about  their  accumulated  wisdom,  what’s 
generally  in  the  heads  of  science  or  math 
or  English  teachers,  making  their  stan- 
dards explicit  and  inviting  public  re- 
sponse. I see  it  very  much  as  a communi- 
cation exercise,  a dialogue  will;  the  pub- 
i lie  to  gain  consensus.  I don’t  sec  it,  as 
some  folks  do,  as  an  end  in  itself,  or  a 
final  act  that  will  survive  forever.  I sec  it 


as  evolutionary,  changing,  dynamic,  and 
engaging  all  those  affected. 

Russell  Is  there  anything  emerging 
now  in  education  that  causes  you  to  fear 
the  latter  possibility,  that  the  benchmarks 
we  have  been  developing  will  become 
fixed,  rigid  standards? 

Archer  I see  some  evidence  with  a 
minority  of  teachers  who  are  not  used  to 
the  public  nature  of  this  dialogue  becom- 
ing defensive,  retreating  into  a discipline 
and  relying  on  internal  standards  that  they 
treat  as  gospel.  I think  we  need  a program 
to  dissuade  the  public  that  benchmarks 
are  a form  of  standardized  test.  As  the 
recent  OISE  Survey  of  Public  Attitudes  to 
Education  showed,  there  is  that  confu- 
sion. I don’t  know  how  to  avoid  it  but  I 
think  that  is  one  of  the  difficulties. 
Benchmarks  set  a priority;  they  start  from 
what  students  can  do  in  a certain  context, 
as  opposed  to  what  they  should  be  doing. 
And  the  sequence  is  important.  You  can’t 
really  anticipate  or  put  predictors  into  it 
until  you  observe  what  can  be  done  in  a 
certain  context  with  a certain  bunch  of 
kids  at  a certain  level.  Benchmarks  pro- 
vide a platform,  a basis  for  better  kinds  of 
testing,  as  opposed  to  being  a test  in  and 
of  themselves.  I think  that  is  one  of  their 
distinguishing  — and  most  valuable  — 
features. 

Russell  Maybe  we  can  close  by  offer- 
ing some  advice  to  teachers  who  are  con- 
cerned about  the  role  they  might  play  in 
developing  and  implementing  assessment 
activities. 

Travers  Teachers  have  the  day-to-day 
experience  of  working  with  youngsters, 
and  so  they  are  uniquely  positioned  to 
have  a lot  of  input  into  the  kinds  of 
assessment  activities  that  are  being  used. 
But  I think  it’s  up  to  folks  like  us  to  find 
ways  to  involve  teachers  in  this  process. 
It’s  far  too  important  to  leave  to  the  pro- 
fessional test-makers.  We  allude,  as  we 
have  in  this  conversation,  to  the  impor- 
tance of  assessment  activities,  whatever 
they  arc,  being  an  integral  part  of 
instruction.  I think  unless  the  teachers 
themselves  have  been  involved,  it’s  not 
realistic  to  talk  about  assessment  as 
being  integral.  In  other  words,  if  we 


don’t  do  that,  we’re  teaching  today  and 
testing  tomorrow,  and  never  the  twain 
shall  meet.  I think  that’s  fatal  in  terms  of 
any  effective  assessment. 

Archer  I think  the  key,  and  it  may  not 
be  as  pursuable  as  it  was  in  the  past,  is 
appropriate  in-service.  There’s  lots  of 
leadership  at  the  board  level  around  the 
province.  And  the  Ministry  is  doing 
some  yeoman  service  in  their  benchmark 
process  which,  to  this  point,  has  been  an 
open  teacher-validated  process.  But  I go 
back  to  Ken’s  comment  about  teachers 
being  best  placed  to  not  only  judge  what 
they’re  seeing  and  how  it  might  be  eval- 
uated but  also  to  recognize  the  need  to 
augment  what  they  can  do  at  the  class- 
room level  with  all  sorts  of  assisted 
materials  and  research  and  strategies.  I 
don’t  think  that’s  done  best  individually. 
I think  it’s  done  collectively  in  an  in- 
service  context  and  I hope  over  the  next 
few  years  we  don’t  shrink  the  opportuni- 
ties to  bring  teachers  together  around 
measurement  and  assessment  issues.  My 
advice  to  teachers  would  be  to  access  as 
much  structured  opportunity  as  is  avail- 
able for  looking  at  these  things  rather 
than  trying  to  sort  it  all  out  alone. 

Summers  I support  what  Bruce  said. 
In  our  experience  with  the  development 
of  Ontario  benchmarks,  we  found  that  the 
most  powerful  part  of  the  process  was 
working  with  teachers,  the  kind  of  really 
excellent  practitioners  who  reflected  on 
their  practice  and  generated  descriptions 
of  standards  that  were  embedded  in  their 
thinking,  in  their  behaviour  in  the  class- 
room, and  then  made  the  description  pub- 
lic. But  the  basic  principle  in  this 
discussion  has  to  be  that  standards  apply 
to  all.  In  trying  to  be  sensitive  to  the 
needs  and  concerns  of  all  students,  there 
is  a tendency,  sometimes,  to  protect  and 
even  prevent  students  from  knowing 
what  the  standard  is  or  what  the  expecta- 
tions are.  I think  that  teachers  do  a super 
job  of  supporting  children  to  grow  and 
improve,  but  being  clear  about  the  stan- 
dard is  definitely  a service  to  students, 
rather  than  a disservice,  as  long  as  we 
help  them  to  grow  and  achieve. 

Russell  I’d  like  to  thank  you  all  for 
participating  in  this  discussion. 
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The  1993  Orbit  Essay  Contest 


Elizabeth  Campbell 

Assistant  Professor 
Faculty  of  Education 
University  of  Toronto 

When  17-  and  18-year-old  secondary 
school  students  were  asked  to  assign  a 
grade  to  their  schools  and  explain  their 
evaluations,  85  of  them  responded  to  this 
year’s  Orbit  contest  by  submitting  essays 
that  were  thoughtful,  sensitive,  humor- 
ous, and  well  written.  The  best  ones  cap- 
tured a sense  of  school  culture  that 
vividly  depicted  life  in  schools  across 
Ontario. 

Congratulations  to  our  winner  — 
David  Collins,  from  Pope  John  Paul  II 
Secondary  School  in  Scarborough  — and 
to  the  runners-up  — Jack  Bennett  at  the 
University  of  Toronto  Schools,  Toronto, 
and  Melissa  Dill  from  Lambton  Kent 
Composite  School  in  Dresden.  They  are 
splitting  $600  donated  by  OISE’s  Alum- 
ni Association. 

It  is  appropriate,  in  this  issue  on  Stan- 
dards/Benchmarks, that  evaluation  is  the 
Essay  Contest  theme.  The  judging  pro- 
cess was  itself  much  like  “a  play  within  a 
play”  in  which  papers  grading  schools 
were,  themselves,  graded.  While  the 
essays  were  assessed  on  the  bases  of 
mechanics,  content,  and  “passion”  or 
style,  the  criteria  used  by  students  to 
grade  their  schools  were  varied. 

Students  used  a variety  of  approaches 
in  their  assessments  of  schools.  Some 
were  largely  critical.  Others  expressed  a 
concern  to  be  fair,  to  take  seriously  their 
evaluative  task  and  provide  a balanced, 
“fair”  judgment.  Some  compared  quali- 
ties across  schools,  while  one  student 
noted,  “I’ve  only  been  to  this  school,  so  I 
can’t  compare  it  with  others.” 

While  using  varying  criteria,  students 
also  reflected  varying  interpretations  of 
what  grades  actually  mean.  For  example, 
some  “great”  schools  were  assigned  a B 


grade,  and  other  “not-so-great”  schools 
were  also  assigned  a B.  For  educators, 
who  also  reflect  similar  variance  in  their 
grading,  what  does  this  imply  about 
shifting  evaluation  standards  across 
classrooms,  schools,  and  school  sys- 
tems? The  subjectivity  of  holistic  grad- 
ing was  further  evident  when  students 
from  the  same  schools  assessed  the 
schools’  merits  and  faults  so  differently 
that  it  required  double-checking  to  con- 
firm that  they  were  discussing  the  same 
schools. 

Despite  these  differences,  educators 
may  be  pleased  to  know  that  most  stu- 
dents, even  given  the  disparity  among 
grade  interpretation,  assigned  their 
schools  A’s  and  B’s;  there  were  a few 
C’s  and  one  failed  grade  given. 

Assessing  Criteria: 

What  Students  Value 

Most  of  the  essays  were  written  from  the 
narrative  first-person  perspective  provid- 
ing an  analysis  of  school  factors;  howev- 
er, one  graded  the  school  through  the 
“eyes”  of  a visiting  friend,  while  another 
directly  discussed  with  the  school  its 
latest  report  card.  Another  student  led  his 
readers  on  a conducted  walking  tour  of 
his  school,  stopping  occasionally  along 
the  way  to  point  out  the  gymnasium,  the 
science  labs,  the  cafeteria, . . . 

Students  provided  many  concrete 
examples  to  support  their  assessments, 
and  common  themes  emerged.  The  aca- 
demic quality  of  the  education  they 
receive  was  seen  to  be  vital  by  students. 
Extracurricular  programs  in  sports, 
music,  and  “special  activities”  were 
important,  as  was  winning  the  district- 
wide school  math  competition.  The  qual- 
ity of  facilities  for  both  athletic  and 
academic  pursuits  was  a significant  fac- 
tor. The  aesthetic  appeal  of  the  schools 
was  of  great  importance  to  students  who 
preferred  a pleasing,  attractive  “home” 


environment  to  an  institutional  building; 
community-like  feelings  of  togetherness, 
friendliness,  and  security  enhanced  the 
good  schools’  ratings.  Especially  impor- 
tant to  most  students  was  the  need  for  the 
school  to  be  safe  and  attentive  to  their 
general  well-being.  One  student  contrast- 
ed her  safe  school  with  a “nightmare” 
school  in  which  students  arm  themselves 
with  knives  for  protection  against  their 
violent  peers.  In  this  sense,  very  few 
questioned  the  authority  structure  of  the 
school;  rules  and  policies  were  seen  to 
enhance  personal  security,  and  most 
favoured  them.  Several  students  men- 
tioned new  “no  tolerance”  policies 
regarding  racism,  drug  use,  and  violence, 
and  they  applauded  their  strict  enforce- 
ment. 

Of  perhaps  the  greatest  importance  in 
evaluating  schools  were  the  interpersonal 
connections  students  make  with  teachers 
and  peers.  Many  focussed  on  the  excep- 
tional teachers  who  make  a difference  in 
their  lives  on  a daily  basis.  Others  spoke 
of  the  social  importance  of  having  good 
friends  and  of  the  need  to  be  aware  of 
issues  such  as  peer  pressure,  “cliquish- 
ness,” and  shyness. 

What  this  year’s  essay  contributions 
revealed  — or,  more  accurately,  rein- 
forced since  I expect  most  of  us  know 
this  from  our  own  school  days  as  well  as 
from  our  work  with  students  — is  the 
fact  that  schools  are  very  important  to 
students  academically,  physically,  social- 
ly, and  emotionally.  Thus,  striving  to 
make  all  schools  A+  schools  is  an  ambi- 
tious but  worthy  aim. 


Dr.  Campbell  recently  won  the  Canadian 
Society  for  the  Study  of  Educational 
Administration’ s Dissertation  of  the  Year 
Award  and  was  named  Teacher  of  the 
Year  at  the  Faculty  of  Education,  Uni- 
versity of  Toronto. 
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— Marisa  Cassano 
Teacher-Librarian 
St.  Nicholas  of  Bari  School 
Metropolitan  Separate  School  Board 


Orbit  is  one  of  the  few  journals  that  blend 
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make  it  a valuable  resource. 

— Clare  Kosnick 
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Orbit  is  an  excellent  catalyst  for  the  professional 
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educational  programs. 
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educators.  Orbit  is  a must  for  educators 
interested  in  the  latest  innovative  programs 
and  ideas  being  put  into  practice. 

— Michael  Fullan 
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University  of  Toronto 
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